Combinatorial Game Theory and Reinforcement Learning in Cumulative Tic-Tac-Toe via Evaluation Functions

Abstract

We introduce cumulative tic-tac-toe, a novel variant of the classic $3\times 3$ tic-tac-toe game in which play continues until the board is completely filled. Each player’s final score is determined by the total number of three-in-a-row sequences they form. Using combinatorial game theory (CGT), we establish that under optimal play, the game is a draw, and we characterize its theoretical properties. To empirically validate and optimize practical play, we develop a reinforcement learning (RL) framework based on temporal-difference (TD) learning, which is enhanced with a domain-informed evaluation function to accelerate convergence. The experimental results show that our triplet-coverage difference (TCD) evaluation function reduces the average number of training episodes by approximately 23.1% compared with a random-initialization baseline, a statistically significant improvement at the 5% significance level. These results demonstrate the efficiency of our CGT–RL approach for cumulative tic-tac-toe and suggest that similar methods may be useful for analyzing related combinatorial games. We also discuss potential analogies in domains such as competitive resource allocation and coalition formation, illustrating how cumulative-scoring games connect abstract game-theoretic ideas to practical sequential decision problems.

1. Introduction

Games are ubiquitous as both entertainment and scholarly objects of investigation. Researchers in many disciplines study both the formal structure of games and how people play them, contributing to the interdisciplinary field of game studies. The continued rise of game studies reflects the cultural importance of games in all formats, media, and interaction modes [1]. A notable example is the multi-volume series of mathematical strategies for a variety of games [2,3,4,5]. Insights from game studies now inform political science [6], psychology [7], sociology [8], linguistics [9], education [10], and computer science [11]. Methodologies span qualitative, quantitative, and mixed-methods approaches [12]. Historically, games of chance shaped the early development of probability theory [13], whereas many enduring puzzles have deep roots in graph theory and other branches of applied mathematics [14].

Two-player games such as Hex [15], chess [16], and checkers [17] pose well-defined challenges governed by explicit rules and social interactions between opponents. Understanding optimal play, which refers to how rational agents should act to maximize their chances of winning, is a central concern. Game theory offers mathematical models of strategic interaction and has shaped research in education [18], behavioral science [19], economics [20], politics [21], and evolutionary biology [22]. Important concepts in game theory include zero-sum games, where players’ payoffs sum to zero, and perfect information games, where every prior move is observed by all players. Classical tools such as minimax search and backward induction remain fundamental for analyzing such games.

However, traditional game theory often addresses settings with incomplete information, where key details are hidden from the players. For example, an iterative algorithm for computing an exact Nash equilibrium for two-player zero-sum games with imperfect information has been proposed [23]. Combinatorial game theory (CGT) complements this perspective by focusing on two-player, zero-sum, perfect-information games of pure skill, whose outcomes are restricted to win, loss, or draw. Several benchmark games have been completely solved via CGT techniques [24]. However, most board games remain theoretically challenging because the exponential growth in the number of possible move sequences renders exhaustive case analysis computationally infeasible [25]. Contemporary approaches, therefore, turn to modern machine learning methods, especially reinforcement learning, for tractable approximations.

Reinforcement learning (RL), one of the three core branches of machine learning alongside supervised and unsupervised learning, studies how agents learn to act in dynamic environments via trial and error [26]. RL has advanced in autonomous driving [27], career planning [28], robotics [29], neuroscience [30], quantitative finance [31], computational applied mathematics [32], and many other domains [33]. In recent years, RL methods have achieved superhuman performance in many games by combining tabular techniques, such as dynamic programming (DP), Monte Carlo (MC), and temporal-difference (TD) learning, with scalable function approximation [34]. These algorithms yield fast, high-accuracy estimates even in large state spaces once they are considered intractable.

Tic-tac-toe typifies a two-player, zero-sum, deterministic, perfect-information game. Players alternately mark empty cells on a $3\times 3$ grid, aiming to align three symbols horizontally, vertically, or diagonally. If all nine cells are filled without a line, the game is a draw, guaranteed by optimal play. Despite its simplicity, tic-tac-toe underpins classroom demonstrations in computer science [26], mathematics education [35], and game theory [36]. Because tic-tac-toe is fully solved theoretically, researchers routinely use it to test new algorithms and evaluate decision-making under uncertainty [37]. In particular, RL algorithms have been successfully applied to tic-tac-toe [38,39,40,41,42]. Building on these results, RL solutions for tic-tac-toe have also been adapted to challenges such as enabling hypermedia agents to perform transfer learning in web environments [43], exploring chain-of-thought reasoning in large language models [44], robotic control tasks [45], and integrating language understanding with gameplay [46].

Numerous variants of tic-tac-toe broaden the original rules, altering board size, introducing imperfect information, or modifying victory conditions. For example, optimal strategies for variants of the dice game Pig have been analyzed using RL [47,48]. These derivatives matter for three reasons. First, they furnish tractable models of real-world scenarios where sequential, irreversible choices accumulate toward long-term rewards. Second, they reveal mathematical structures developed for generalization. Third, progress on complex variants often translates into advances in artificial-intelligence capabilities with practical impact far beyond games [49].

This paper introduces cumulative tic-tac-toe, a new variant inspired by practical application scenarios. We first formalize the game and discuss its potential relevance to practical decision problems. Next, we analyze the theoretical properties within the CGT framework and cast the game as a Markov decision process (MDP) to facilitate RL-based verification. We integrate a standard tabular RL algorithm with a domain-informed evaluation function that encodes expert priors to accelerate learning. Extensive experiments compare convergence with and without our evaluation function, demonstrating a notable improvement in training efficiency. We conclude by discussing implications for future research.

In summary, the main contributions of this paper are the following. We introduce cumulative tic-tac-toe, a scoring variant of $3\times 3$ tic-tac-toe in which play continues until the board is filled and payoffs are determined by the total number of completed three-in-a-row sequences. We analyze cumulative tic-tac-toe within a combinatorial game-theoretic framework and prove that, under optimal play, the game is a draw (Theorems 1 and 2). We then formulate cumulative tic-tac-toe as a finite MDP and develop a practical one-step TD learning procedure seeded with a novel CGT-motivated evaluation function, the triplet-coverage difference (TCD). While one-step TD is a standard RL technique, the novel element of this paper is the CGT-guided initialization: we seed the tabular value table using the TCD evaluation function, thereby leveraging combinatorial structure to accelerate TD learning in cumulative tic-tac-toe. Our experiments show that TCD initialization reduces episodes-to-convergence by approximately 23.1% compared with random initialization (one-sided Student’s t test, p-value = 0.0203), while preserving the draw outcome predicted by CGT. Finally, we discuss how this CGT–RL pipeline can inform the study of other cumulative-scoring decision problems and outline directions for scaling the approach to larger or more complex games.

2. Cumulative Tic-Tac-Toe

In this section, we first formalize the mechanics of cumulative tic-tac-toe by detailing its rules and scoring conventions. We then demonstrate its broader relevance by mapping the game’s cumulative-scoring framework onto six representative real-world decision problems drawn from engineering, economics, and the social sciences. Finally, we consider the variant’s extensibility to higher-dimensional boards and discuss how deviations from perfect play can lead to empirical outcomes that differ from the game’s theoretical determinacy.

2.1. Rules of the Game

Numerous extensions of classical tic-tac-toe have been proposed, including natural generalization to $n\times n=n^2$ boards and more generally to $n\times \dots \times n=n^d$ hypercubes [37,50,51,52]. Another notable variant is quantum tic-tac-toe, which is based on concepts from quantum physics [53].

Cumulative tic-tac-toe modifies the standard rules by allowing play to continue until all $3\times 3=9$ cells are filled rather than ending when the first three-in-a-row sequence, referred to as the first triplet, is formed. The outcome is determined by the total number of triplets secured by each player throughout the play. Thus, instead of a binary win/loss decision at the moment a triplet is formed, players compete to accumulate as many triplets as possible over the entire course of play. Each mark placed may contribute to multiple overlapping triplets, increasing the strategic depth of the game.

Importantly, cumulative tic-tac-toe preserves the spatial structure of the classical version, retaining the eight canonical triplets: three rows, three columns, one diagonal, and one antidiagonal.

Table 1. A detailed table summarizing the similarities and differences between the classic tic-tac-toe and cumulative tic-tac-toe.

Aspect	Classic Tic-Tac-Toe	Cumulative Tic-Tac-Toe
Termination	Play ends immediately upon triplet or full board.	Play continues until all cells are occupied.
Win Condition	First to make a completed triplet wins; if the board fills with no triplet, it is a draw.	Player with the highest total number of triplets wins; equal total yields in a draw.
Play Length	At most nine moves; may end earlier when someone wins.	Exactly nine moves (all cells filled).
Outcome Possibilities	Win, draw, or lose.	Win, draw, or lose.
Strategy Complexity	Relatively simple; the game is solved, and optimal play leads to a draw.	Higher complexity, as each placement can contribute to multiple triplets; potentially richer mid- and endgame decisions.
Game Solved Status	Solved as a draw under optimal play.	Not yet solved; an explicit optimal strategy remains unknown.

provides a detailed comparison between the classic tic-tac-toe and cumulative tic-tac-toe. We will examine further theoretical insights and properties of this variant in the context of CGT in a later section.

Cumulative tic-tac-toe introduces more strategic depth than the standard $3\times 3$ game while remaining fully analyzable on the small board. The rules can be extended to larger boards, potentially allowing exploration of more complex sequential decision processes. Thus, cumulative tic-tac-toe may serve as a tractable model for studying strategic behavior with cumulative scoring.

2.2. Real-World Scenarios

Cumulative tic-tac-toe features sequential decisions with cumulative payoffs that can arise in various strategic settings. We outline six example scenarios (Figure 1) to illustrate potential analogies in engineering, economics, and social-science applications. These illustrative examples suggest how similar decision dynamics may appear in practice, though they are meant as analogies rather than direct implementations. Cumulative tic-tac-toe serves as a minimalist abstract model that isolates a specific structural property common to these complexities so that we can rigorously analyze the combinatorial nature of cumulative scoring in a controlled, fully observable environment.

Competitive resource allocation (e.g., dynamic spectrum access [54,55]) involves agents sequentially claiming resources that remain occupied for a duration. The cumulative throughput depends on these sequential allocations, analogous to accumulating triplets in the game. In such settings, strategies must balance immediate resource gains against long-term positioning, similar to the trade-offs in cumulative tic-tac-toe.

Iterative coverage tasks (e.g., unmanned aerial vehicle mapping [56]) involve agents sequentially securing grid cells to maximize total coverage. Each secured cell contributes to the overall payoff, much like each triplet contributes to a player’s score in cumulative tic-tac-toe. Effective coverage strategies must balance immediate area gains with preserving future coverage opportunities, reflecting trade-offs analogous to those in the game.

Sequential coalition-building (e.g., forming majority coalitions across successive votes [57]) involves accumulating influence over rounds to meet decision thresholds. Overlapping coalition possibilities can arise as actors join multiple coalitions, analogous to how central cells in the cumulative tic-tac-toe grid participate in multiple triplets. The process of sequentially assembling these coalitions resembles building multiple triplets in the game.

Adversarial spatial games like Go or Hex involve long-term strategic control on a board [58,59]. Cumulative tic-tac-toe is a simpler, fully observable analog that preserves key spatial–strategic dynamics. It provides a tractable testbed for developing and evaluating value-based RL methods or curricula before applying them to more complex games.

Iterative auctions (e.g., spectrum auctions [60]) involve bidders placing sequential bids to assemble complementary bundles. Each bid influences future opportunities, similar to how placing a marker in the game opens or closes potential triplets. The problem of ordering bids to assemble winning bundles is analogous to choosing moves that maximize triplet formation in cumulative tic-tac-toe.

Sequential market-entry models involve firms iteratively expanding into new markets or related sectors [61,62], creating overlapping strategic positions. Building these complementary activities is analogous to forming multiple triplets in cumulative tic-tac-toe, where each move contributes to several potential scoring lines. As in the game, each expansion shapes future opportunities, and success depends on the sequence of moves.

These six scenarios demonstrate how the cumulative tic-tac-toe paradigm captures long-term reward dynamics in strategic decision problems. This framework can be extended to higher-dimensional settings or adapted to reflect more complex real-world conditions. On the other hand, because these real-world scenarios are often too complex for an exact combinatorial solution, practitioners rely on heuristics. The use of RL mirrors how real-world agents learn to approximate optimal strategies in these cumulative-payoff environments when exact calculation is impossible.

We also note that classical game-theoretic analysis typically assumes perfect rationality. That is, every participant is fully informed, never makes errors, and, therefore, selects the strategy that maximizes their eventual payoff in every contingency [20,36]. Real decision-makers are fallible. Slips, misperceptions, and bounded cognitive resources frequently derail optimal play. For example, in classic tic-tac-toe, such departures open the door for the second player to capture a win, even though under flawless play, the best the second player can guarantee is a draw [37]. These empirical upsets do not contradict the formal theoretical results in CGT. They simply underline the behavioral gap between theory and practice.

3. Empirical Methodology

In game theory, an extensive-form representation is a tree-based structure that graphically models sequential decision-making processes [20,36]. The nodes of the tree represent decision points, such as a player’s move, and branches correspond to available actions. These diagrams conventionally depict nodes as solid circles and branches as connecting arrows. Given that cumulative tic-tac-toe is a sequential game, it is natural to illustrate its early moves via such a representation. A complete game tree captures all possible plays, with each branch encoding a unique sequence of decisions. A partial game tree captures only the possible plays at a certain decision point. Computer science has also explored the discrete structure and algorithms associated with game trees [63,64].

While traditional game theory suggests techniques such as backtracking or backward induction to analyze game trees, these approaches become infeasible for complex games due to exponential time requirements. Simulating random play is straightforward, but deriving optimal play is computationally intractable and typically lacks empirical grounding. RL offers a data-driven alternative for approximating or determining optimal strategies. Even when the outcome class (e.g., draw or first-player win) of a game is theoretically established, uncovering optimal strategies remains an open problem. For instance, although Hex is known to be a first-player win [65,66], a complete winning strategy is still unknown [37]. By allowing both players to engage in repeated play through an RL algorithm, we can empirically infer optimal decisions at each decision point. This exemplifies the synergy between CGT and RL in the study of novel game variants.

3.1. Combinatorial Game Theory

Building directly on the extensive-form tree, CGT provides the mathematical foundation for analyzing two-player zero-sum games of perfect information and no chance elements. The CGT methodology can be broadly divided into two areas: the theory of impartial games (e.g., Nim) and the theory of partizan games (e.g., tic-tac-toe) [2,37,67]. In impartial games, all players have the same available moves from any given position. These games are often decomposed into independent subgames, and their values are analyzed using the Sprague–Grundy theorem, which assigns nimbers, values on the basis of the single-heap version of Nim, to characterize outcomes and simplify the combinations of games [67,68,69].

Cumulative tic-tac-toe is inherently a partizan game, as the available moves and resulting outcomes depend on which player acts. Consequently, it cannot be analyzed via the Sprague–Grundy framework and instead falls within the scope of partizan or positional games, a subclass of combinatorial games.

In the CGT formulation, a positional game involves two players, Player 1 (first mover) and Player 2, who alternately claim positions on a finite board [52]. The game is governed by a defined set of rules, including an initial position and deterministic transitions based on each move. A move entails selecting an unclaimed position on the board, and a play is a complete sequence of alternating moves terminating in a terminal state. The game’s outcome is fully determined by this terminal state according to fixed rules. Importantly, these games exhibit perfect information, a core assumption in CGT.

Within this framework, the objective is to identify optimal strategies: strategies that allow a player to force a win or guarantee a draw in the absence of a winning path. However, calculating such strategies directly is typically infeasible because of the exponential size of the game tree. Fortunately, the RL approach presented after CGT offers a practical path to discovering these strategies through a data-driven approximation.

3.1.1. Formal Hypergraph Representation

We model cumulative tic-tac-toe as a finite hypergraph game $\mathcal{G}=(V,\mathcal{F})$, where

$$V=\{1,2,\dots ,9\}$$

labels the nine board cells, and 

$$\mathcal{F}=\left\{\{1,2,3\},\{4,5,6\},\{7,8,9\},\{1,4,7\},\{2,5,8\},\{3,6,9\},\{1,5,9\},\{3,5,7\}\right\}\subset 2^V$$

is the collection of the eight triplets (rows, columns, and diagonals) relevant to determining final scores at terminal states.

A complete play of cumulative tic-tac-toe can be viewed as a sequence of moves

$$(x_1,y_1,x_2,y_2,x_3,y_3,x_4,y_4,x_5),$$

where $x_i$ (for $i=1,2,3,4,5$) is the ith move by Player 1 and $y_j$ (for $j=1,2,3,4$) is the jth move by Player 2. Similarly, we index the decision times by $t=0,1,\dots ,8$, so that at time t, exactly t moves have already been made. Then, Player 1 moves when t is even ($t=0,2,4,6,8$), choosing $x_{k+1}$ at $t=2k$, and Player 2 moves when t is odd ($t=1,3,5,7$), choosing $y_{\mathcal{\ell }}$ at $t=2\mathcal{\ell }-1$.

We denote by $\mathcal{H}$ the set of all finite sequences of distinct elements of V, i.e.,

$$\mathcal{H}=\left\{(v_1,v_2,\dots ,v_t):t\ge 0,v_i\in V,v_i\ne v_j,i\ne j\right\},$$

and for each $t\in \{0,1,\dots ,8\}$, let $h_t\in \mathcal{H}$ be the history (sequence) of length t just before the decision at time t. In particular,

$$h_0=\left(\right),h_1=\left(x_1\right),h_2=(x_1,y_1),h_3=(x_1,y_1,x_2),\dots ,h_8=(x_1,y_1,x_2,y_2,x_3,y_3,x_4,y_4).$$

Thus, Player 1 observes $h_{2k}$ when making their $(k+1)$th move at time $2k$ ($k=0,1,2,3,4$), and Player 2 observes $h_{2\mathcal{\ell }-1}$ when making their ℓth move at time $2\mathcal{\ell }-1$ ($\mathcal{\ell }=1,2,3,4$).

We define the special time $t=9$ to mark the conclusion of the game. Although the decision times run from $t=0$ to $t=8$, the terminal time $t=9$ serves as a bookkeeping point at which the result of the full play is evaluated. At the terminal history $h_9=(x_1,y_1,\dots ,x_4,y_4,x_5)$, one defines

$${\mathcal{T}}_1=\left\{e\in \mathcal{F}:e\subset \{x_1,x_2,x_3,x_4,x_5\}\right\},{\mathcal{T}}_2=\left\{e\in \mathcal{F}:e\subset \{y_1,y_2,y_3,y_4\}\right\},$$

i.e., ${\mathcal{T}}_1$ (${\mathcal{T}}_2$) is the collection of triplets fully occupied by Player 1 (Player 2). These sets determine the final play outcome under the cumulative scoring rule.

A pure strategy for a player is a function

$$Str:{\mathcal{H}}_{\mathrm{turn}}\to V,$$

where ${\mathcal{H}}_{\mathrm{turn}}\subset \mathcal{H}$ is the subset of histories at which it is that player’s turn to move. The requirement is that for each $h\in {\mathcal{H}}_{\mathrm{turn}}$,

$$Str\left(h\right)\in V\setminus \left\{\mathrm{entries}\mathrm{of}h\right\},$$

so that $Str\left(h\right)$ is a legal move not already played.

A particularly useful type of pure strategy in CGT is a pairing strategy. A pairing is a bijection $\rho :M\to N$, where $M,N\subset V$, $\left|M\right|=\left|N\right|=\lfloor \left|V\right|/2\rfloor =4$, and $M\cap N=\varnothing$. A pairing strategy $PStr(·)$ then replies to an opponent’s move by playing the paired cell, subject to legality. To formalize it, at time t, suppose that the opponent’s most recent move (in history $h_t$) is $v\in V$. If $v\in M$, reply with $\rho \left(v\right)$; if $v\in N$, reply with ${\rho }^{-1}\left(v\right)$; if the pairing pair is already occupied or $t=0$ (first move), choose arbitrarily among the remaining legal moves. Mathematically, one may define

$$PStr\left(h_t\right)=\left\{\begin{array}{cc}\rho \left(v\right),\hfill & \mathrm{if}\mathrm{the}\mathrm{last}\mathrm{move}v\mathrm{in}{h}_t\mathrm{lies}\mathrm{in}M\mathrm{and}\rho \left(v\right)\notin h_t,\hfill \\ {\rho }^{-1}\left(v\right),\hfill & \mathrm{if}\mathrm{the}\mathrm{last}\mathrm{move}v\mathrm{in}{h}_t\mathrm{lies}\mathrm{in}N\mathrm{and}{\rho }^{-1}\left(v\right)\notin h_t,\hfill \\ \mathrm{any}m\in V\setminus \left\{\mathrm{entries}\mathrm{of}h\right\},\hfill & \mathrm{otherwise}(\mathrm{e}.\mathrm{g}.,\mathrm{if}\mathrm{the}\mathrm{paired}\mathrm{cell}\mathrm{is}\mathrm{occupied}\mathrm{or}\mathrm{at}t=0).\hfill \end{array}\right.$$

This ensures that $PStr\left(h_t\right)$ is always a legal move. Specific versions of pairing strategies fix how the arbitrary choices are made (e.g., random or following a fixed priority).

3.1.2. Play Outcomes

A play of cumulative tic-tac-toe is a full sequence of nine distinct moves, alternating between Player 1 and Player 2, that fills the board. At terminal history $h_9$, the outcome of a play is determined by comparing the numbers of fully occupied hyperedges for each player:

• Player 1 wins if $\left|{\mathcal{T}}_1\right|>\left|{\mathcal{T}}_2\right|$.
• Player 2 wins if $\left|{\mathcal{T}}_1\right|<\left|{\mathcal{T}}_2\right|$.
• Otherwise, $\left|{\mathcal{T}}_1\right|=\left|{\mathcal{T}}_2\right|$, the game is a draw.

Because Player 1 occupies exactly five cells and Player 2 occupies exactly four cells in a complete play,

$$0\le \left|{\mathcal{T}}_1\right|\le 2,0\le \left|{\mathcal{T}}_2\right|\le 1,0\le \left|{\mathcal{T}}_1\right|+\left|{\mathcal{T}}_2\right|\le 2.$$

Consequently, the pair $\left(\right|{\mathcal{T}}_1|,|{\mathcal{T}}_2\left|\right)$ can take exactly five possible values over all plays (namely, $(0,0)$, $(1,0)$, $(2,0)$, $(0,1)$, $(1,1)$), and the above rules partition these values into play outcomes.

3.1.3. Game Determinacy

Before analyzing specific strategies, we first establish that cumulative tic-tac-toe is a determined game: under optimal play, exactly one of the three outcomes (Player 1 wins, Player 2 wins, or a draw) must occur [70]. In particular, we use a classical strategy-stealing argument [71,72,73] to eliminate the possibility of a forced win for Player 2. It follows that Player 1 can guarantee at least a draw. Theorem 1 formalizes this determinacy.

Theorem 1.

Consider cumulative tic-tac-toe on the finite hypergraph $\mathcal{G}=(V,\mathcal{F})$. Then, the game is determined in the sense that either Player 1 has a winning strategy or both players have drawing strategies.

Proof. 

The proof appears in Appendix A.1.    □

3.1.4. Game Outcomes

We refine our understanding of determinacy by classifying the game according to stronger notions of forced outcomes and the existence or nonexistence of a single universal pairing strategy that guarantees a draw. Informally, even if a game is a draw under optimal play, one may ask whether there is a single pairing function $\rho$ such that the pairing strategy ensures a draw for whichever player employs it, regardless of how the opponent plays (subject to legality). We introduce the following outcome classes for cumulative tic-tac-toe:

• Player 1 trivial win: Regardless of how both players behave, every play ends with $\left|{\mathcal{T}}_1\right|>\left|{\mathcal{T}}_2\right|$. Equivalently, every play is a Player 1 win.
• Player 1 forced win without draw possibility: Under optimal play by both, every play ends with $\left|{\mathcal{T}}_1\right|>\left|{\mathcal{T}}_2\right|$. However, there exist suboptimal plays where $\left|{\mathcal{T}}_1\right|<\left|{\mathcal{T}}_2\right|$ but never $\left|{\mathcal{T}}_1\right|=\left|{\mathcal{T}}_2\right|$. Equivalently, Player 1 has a forced win, and no drawing play can occur under every play.
• Player 1 forced win but draws possible if suboptimal: Player 1 has a winning strategy, so under optimal play, we always have $\left|{\mathcal{T}}_1\right|>\left|{\mathcal{T}}_2\right|$, yet there exist suboptimal plays where $\left|{\mathcal{T}}_1\right|\le \left|{\mathcal{T}}_2\right|$.
• Strong draw (no single universal pairing strategy draw): Under optimal play by both, the result is always a draw $\left|{\mathcal{T}}_1\right|=\left|{\mathcal{T}}_2\right|$, but there is no single pairing function $\rho$ such that whichever player uses the corresponding pairing-based reply strategy can force a draw against every possible opponent reply.
• Pairing strategy draw: Under optimal play by both parties, the game is always a draw, and there exists a single pairing function $\rho$ such that if Player 1 (Player 2) employs the pairing reply strategy based on $\rho$, then irrespective of how the opponent plays legally, the result is a draw. In other words, the same $\rho$ yields a draw-forcing strategy for either role.

According to earlier analyses of play outcomes, cumulative tic-tac-toe is not in the first two classes. We now locate it in one of the latter three classes and show the following:

Theorem 2.

Cumulative tic-tac-toe is a strong draw.

Proof. 

The proof appears in Appendix A.2.    □

From the CGT perspective, the fact that no single pairing strategy suffices shows that deriving an explicit optimal policy by pure combinatorial arguments is highly nontrivial. These combinatorial observations identify specific structural features that correlate strongly with outcome: in particular, the number and distribution of open triplets determine the player’s options for future triplet completion. We transfer this CGT insight into the RL pipeline by designing the TCD evaluation function. Concretely, TCD quantifies the difference in open triplet counts favorable to the current player and is used to initialize the tabular value function. This CGT–RL mapping preserves the theoretical understanding of which positions are strong while enabling data-driven policy discovery in spaces where an exhaustive combinatorial solution is onerous.

3.2. Reinforcement Learning

To analyze cumulative tic-tac-toe through both CGT and RL, we must express the game in representations familiar to each framework. In the RL setting, we represent the game as a finite MDP, allowing CGT concepts to map cleanly onto RL notation and enabling data-driven discovery of optimal play. With the game cast as an MDP, we introduce one-step TD learning as our core value estimation method. To accelerate convergence, we present domain-informed evaluation functions, particularly the TCD, and detail our TD algorithm, which seeds its value table with the heuristic. While one-step TD itself is a standard RL technique, our contribution lies in a CGT-guided initialization: we propose and evaluate the TCD evaluation function to seed the tabular value function, thereby leveraging structural information from CGT to accelerate TD learning in cumulative tic-tac-toe. Finally, we discuss several alternative evaluation functions and explain how they can further improve learning efficiency.

3.2.1. Markov Decision Process

An MDP formalizes sequential decision-making under uncertainty. An agent interacts with an environment over discrete time steps. At each step, the agent observes the current state, selects an action, receives a numerical reward (possibly contingent on the chosen action), and transitions to a new state.

These interactions generate a sequence, or trajectory, of states, actions, and rewards. If a trajectory ends in a designated terminal state, the sequence resets, and a new trajectory begins from the initial state. A complete sequence from the start state to the terminal state constitutes an episode.

Crucially, each action influences not only the immediate reward but also future states and, hence, future rewards. The agent, therefore, seeks to maximize the cumulative reward over the entire episode rather than the reward at the next step alone. To do so, it follows a policy, a mapping from states to probability distributions over available actions. Given a policy, the state-value function assigns to every state the expected return (cumulative reward) obtained by following that policy until termination.

Different policies induce different state-value functions. The objective is to approximate the optimal state-value function, which is defined by acting optimally at every decision point. Estimating these values allows the agent to attribute long-term consequences to individual decisions and converge on strategies that maximize the expected return.

3.2.2. Formal MDP Representation

We now cast cumulative tic-tac-toe as a two-agent zero-sum MDP in which the control alternates between Player 1 and Player 2. In practice, one may train a single self-play algorithm, but it is equivalent to viewing one player as the agent and the other as part of the environment.

States and Actions

Let the discrete decision times be $t=0,1,\dots ,8$, where, at time t, exactly t cells are occupied. We similarly define the special time $t=9$ as the terminal time. Denote by

$$S_t=(s_t^1,s_t^2,\dots ,s_t^9)\in \mathcal{S}={\{-1,0,1\}}^9$$

the board configuration at time t, using the encoding

$$s_t^i=\left\{\begin{array}{cc}1,\hfill & \mathrm{cell}i\mathrm{occupied}\mathrm{by}\mathrm{Player}1,\hfill \\ -1,\hfill & \mathrm{cell}i\mathrm{occupied}\mathrm{by}\mathrm{Player}2,\hfill \\ 0,\hfill & \mathrm{cell}i\mathrm{unoccupied}.\hfill \end{array}\right.$$

The turn to move is implicit: if ${\sum }_{i=1}^9{s}_t^i=0$, it is Player 1’s turn; if ${\sum }_{i=1}^9{s}_t^i=1$, it is Player 2’s turn; terminal states satisfy ${\sum }_{i=1}^9\left|s_9^i\right|=9$.

At each nonterminal $S_t$, the legal action set is

$$\mathcal{A}\left(S_t\right)=\{i:s_t^i=0\}\subseteq \{1,2,\dots ,9\},$$

and choosing $A_t=i\in \mathcal{A}\left(S_t\right)$ yields

$$S_{t+1}=Update(S_t,i),$$

where $Update(S_t,i)$ sets $s_{t+1}^i=1$ if Player 1 moves or $-1$ if Player 2 moves, and leaves other cells unchanged. Since there is no chance, each transition is deterministic:

$$\mathbb{P}\left(S_{t+1}\right|S_t,A_t)\in \{0,1\},$$

where $\mathbb{P}(·|S_t,A_t)$ denotes the conditional probability measure over all possible successor states given the current state and action.

Rewards

All intermediate rewards $R_t=0$ for $t<9$. At the terminal step $t=9$, we assign the zero-sum payoffs for Player 1 and Player 2

$$R_9^1=\left\{\begin{array}{cc}+1,\hfill & \left|{\mathcal{T}}_1\right|>\left|{\mathcal{T}}_2\right|,\hfill \\ 0,\hfill & \left|{\mathcal{T}}_1\right|=\left|{\mathcal{T}}_2\right|,\hfill \\ -1,\hfill & \left|{\mathcal{T}}_1\right|<\left|{\mathcal{T}}_2\right|,\hfill \end{array}\right.\mathrm{and}{R}_9^2=-R_9^1.$$

Thus, a complete episode yields

$$S_0,A_0,S_1,A_1,\dots ,S_8,A_8,(R_9^1,R_9^2),S_9.$$

Policies and Value Functions

A policy (possibly stochastic) for Player 1 is ${\pi }^1:\mathcal{S}\to \Delta \left(\mathcal{A}\right)$, where $\Delta \left(\mathcal{A}\right)$ denotes the set of all probability distributions over the legal action set $\mathcal{A}$, and similarly Player 2 has ${\pi }^2$. We combine them into a joint policy

$$\pi \left(A_t\right|S_t)=\left\{\begin{array}{cc}{\pi }^1\left(A_t\right|S_t),\hfill & \mathrm{if}{\sum }_{i=1}^9{s}_t^i=0,\hfill \\ {\pi }^2\left(A_t\right|S_t),\hfill & \mathrm{if}{\sum }_{i=1}^9{s}_t^i=1.\hfill \end{array}\right.$$

Under $\pi$, the state-value for Player 1 is

$$v_{\pi }^1\left(S_t\right)={\mathbb{E}}_{\pi }\left(R_9^1\right|S_t),$$

where the expectation ${\mathbb{E}}_{\pi }(·|S_t)$ is taken over all trajectories generated by following $\pi$ from state $S_t$ until terminal time $t=9$. For Player 2, $v_{\pi }^2\left(S_t\right)=-v_{\pi }^1\left(S_t\right)$ because cumulative tic-tac-toe is a zero-sum game. Since all the rewards occur at $t=9$, these satisfy the deterministic Bellman recursion

$$v_{\pi }^1\left(S_t\right)=\left\{\begin{array}{cc}\sum _{a\in \mathcal{A}\left(S_t\right)}{\pi }^1\left(a\right|S_t)v_{\pi }^1(Update(S_t,a)),\hfill & \mathrm{if}{\sum }_{i=1}^9{s}_t^i=0,\hfill \\ \sum _{b\in \mathcal{A}\left(S_t\right)}{\pi }^2\left(b\right|S_t)v_{\pi }^1(Update(S_t,b)),\hfill & \mathrm{if}{\sum }_{i=1}^9{s}_t^i=1.\hfill \end{array}\right.$$

Optimality

Because the game is zero-sum, there exists a saddle-point minimax solution, which is based on CGT. Denote the optimal value for each player by

$$v_{*}^1\left(S_t\right)=\underset{{\pi }^1}{max}\underset{{\pi }^2}{min}{v}_{\pi }^1\left(S_t\right)\mathrm{and}{v}_{*}^2\left(S_t\right)=-v_{*}^1\left(S_t\right).$$

The Bellman optimality equation for Player 1 is

$$v_{*}^1\left(S_t\right)=\underset{a\in \mathcal{A}\left(S_t\right)}{max}\underset{b\in \mathcal{A}(Update(S_t,a))}{min}{v}_{*}^1(Update(Update(S_t,a),b)).$$

Solving this MDP by RL yields policies ${\pi }_{*}^1$ and ${\pi }_{*}^2$ that achieve the minimax value. This computational approach complements the combinatorial game-theoretic guarantee that the game is a draw under optimal play, while providing an explicit procedure to approximate those optimal policies.

3.2.3. Temporal-Difference Learning

Classical solutions to RL problems, most notably DP, can, in principle, compute exact optimal values by solving the Bellman equations for $v_{\pi }$. In practice, this demands a complete model of the environment, including the opponent’s transition probabilities at every decision node, and incurs an exponential search cost that quickly becomes prohibitive. An alternative is to learn from experience. In MC methods, we estimate $v_{\pi }\left(s\right)$ by averaging the returns observed on every first visit to state s while following policy $\pi$ and denote this empirical estimate by $V\left(s\right)$. Owing to the strong law of large numbers, $V\left(s\right)$ converges almost surely to $v_{\pi }\left(s\right)$ as the number of first visits tends to infinity. However, MC updates occur only when each episode terminates.

TD learning combines the online updates of MC with the bootstrapping of DP. After every step, TD methods adjust the current state’s value toward the estimated value of its successor, thus learning directly from interaction without a full model. Because of its balance of efficiency and accuracy, we adopt a one-step TD method for cumulative tic-tac-toe. Function approximation methods in RL can also be readily applied when the state or action space is too large or continuous in complex, real-world scenarios.

Initialization

With a tabular representation, we store two separate estimated value functions, one per player:

$$V^1,V^2:\mathcal{S}\to [-1,1],$$

where $V^1\left(s\right)$ denotes the estimated terminal payoff for Player 1 starting from state s, and $V^2\left(s\right)$ denotes the estimated terminal payoff for Player 2. In the RL literature, one initializes every nonterminal state to an arbitrary value, but we introduce a more informative heuristic that accelerates learning.

Action Selection

During training, both players repeatedly simulate play. At each move, they choose among the legal actions:

• Exploitation: select a greedy action whose successor state currently has the highest estimated value, with ties broken arbitrarily.
• Exploration: with probability $ϵ$, select a random action, thereby refining estimates for suboptimal moves that may conceal tactical traps.

Let $\left|\mathcal{A}\right(s\left)\right|$ denote the number of legal moves in state s. Then, under an $ϵ$-greedy policy, the greedy move is chosen with probability

$$1-ϵ+{\displaystyle \frac{ϵ}{\left|\mathcal{A}\right(s\left)\right|}},$$

and each non-greedy move with probability $ϵ/\left|\mathcal{A}\right(s\left)\right|$.

Value Update

Because all intermediate rewards vanish, one-step TD updates simplify. When Player 1 acts in $S_t$ and yields successor $S_{t+1}$, we update Player 1’s value estimate of $S_t$ by

$$V^1\left(S_t\right)\leftarrow V^1\left(S_t\right)+\alpha \left[V^1\left(S_{t+1}\right)-V^1\left(S_t\right)\right],$$

and symmetrically for Player 2,

$$V^2\left(S_t\right)\leftarrow V^2\left(S_t\right)+\alpha \left[V^2\left(S_{t+1}\right)-V^2\left(S_t\right)\right],$$

where $\alpha \in (0,1]$ is a step-size parameter and at terminal states, we set

$$V^1\left(s_{\mathrm{terminal}}\right)=R_9^1\in \{+1,0,-1\}\mathrm{and}{V}^2\left(s_{\mathrm{terminal}}\right)=-V^1\left(s_{\mathrm{terminal}}\right)$$

so that terminal estimates equal the true terminal returns.

3.2.4. Evaluation Functions

The game tree of cumulative tic-tac-toe is already very large, not to mention the future variants of such a game and their potential applications in practice. Various compression techniques exist, most notably merging board configurations that are equivalent under symmetry, but different move orders can reach the same configuration, so perfect deduplication requires nontrivial bookkeeping, and the resulting trick is seldom portable to other games.

A more general remedy is to seed learning with an evaluation function. Heuristics using evaluation functions are common in chess engines and other supervised settings [74,75,76,77], and their systematic use in RL remains comparatively underexplored [49]. An evaluation function assigns an informative initial value, instead of an arbitrary value, to every nonterminal state, guiding exploration and accelerating convergence.

Because unsolved games lack closed-form value formulas, the shape of an evaluation function is chosen empirically. One inserts a candidate score, probabilistic or ordinal, into the RL loop and measures performance. For instance, a state might be initialized with the estimated win probability under random play or with a signed advantage score where positive values favor the current player and negative values favor the opponent. Note that the design of evaluation functions depends heavily on various factors such as the game-theoretical properties, MDP formulation, and appropriate RL solution methods.

Triplet-Coverage Difference

CGT analysis shows that configurations with many open triplets, from either player’s perspective, are intrinsically strong. Assume we are at a particular board state s. Let X (Y) denote the set of cells occupied by Player 1 (Player 2). If it is Player 1’s turn, define

$$d_{\mathrm{TCD}}\left(s\right)=\left(\mathrm{triplets} \mathrm{containing} \mathrm{no} \mathrm{cell} \mathrm{in} Y\right)-\left(\mathrm{triplets} \mathrm{containing} \mathrm{no} \mathrm{cell} \mathrm{in} X\right).$$

We call $d_{\mathrm{TCD}}\left(s\right)$ the triplet-coverage difference (TCD). A positive $d_{\mathrm{TCD}}\left(s\right)$ means that Player 1 has more ways to complete a triplet in the given board configuration; a negative $d_{\mathrm{TCD}}\left(s\right)$ favors Player 2. From Player 2’s perspective, the evaluation is simply $-d_{\mathrm{TCD}}\left(s\right)$.

For consistency with the MDP we constructed, whose value range is $[-1,1]$, we normalize $d_{\mathrm{TCD}}\left(s\right)$ via its theoretical bounds 4 (best) and $-2$ (worst):

$${\tilde{d}}_{\mathrm{TCD}}\left(s\right)={\displaystyle \frac{d_{\mathrm{TCD}}\left(s\right)-1}{3}}\in [-1,1],$$

so that it can be applied more effectively and efficiently. Therefore, we initialize the two value tables for each nonterminal state s by setting

$$V^1\left(s\right)\leftarrow \left\{\begin{array}{cc}{\tilde{d}}_{\mathrm{TCD}}\left(s\right),\hfill & \mathrm{if}\mathrm{Player}1\mathrm{is}\mathrm{to}\mathrm{move}\mathrm{in}s,\hfill \\ -{\tilde{d}}_{\mathrm{TCD}}\left(s\right),\hfill & \mathrm{if}\mathrm{Player}2\mathrm{is}\mathrm{to}\mathrm{move}\mathrm{in}s,\hfill \end{array}\right.V^2\left(s\right)\leftarrow -V^1\left(s\right).$$

3.2.5. Algorithm

We now describe a one-step TD algorithm that uses TCD as an informative prior for all nonterminal states while assigning each terminal state its exact payoff ($\pm 1$ or 0 depending on $|{\mathcal{T}}_1|$ and $|{\mathcal{T}}_2|$). At each time step, the current player i selects moves according to an $ϵ$-greedy policy derived from the estimated state-value function $V^i$, exploiting the highest-valued successor most of the time but occasionally exploring random actions, and then immediately bootstraps $V^i\left(S\right)$ toward the current estimate $V^i\left(S^{\prime }\right)$ of the successor state. The complete algorithm pseudocode is provided as Algorithm 1, and the corresponding flowchart is shown in Figure A1.

Algorithm 1: One-Step TD with TCD Initialization

3.2.6. Alternative Evaluation Functions

Beyond the basic TCD evaluation function, several refinements can, in principle, further accelerate convergence.

Weighted Triplet-Coverage Difference

The weighted triplet-coverage difference (WTCD) evaluation function is a modified version of TCD. Because the center cell participates in four triplets, it is more valuable than a corner (three triplets) or an edge (two triplets). Let each triplet be weighted by the sum of the weights of its constituent cells (e.g., center $=2$, edges $=1.5$, corners $=1$). Define X and Y similarly as in TCD. The WTCD, if it is Player 1’s turn at state s,

$$d_{\mathrm{WTCD}}\left(s\right)=\left(\mathrm{weighted} \mathrm{triplets} \mathrm{containing} \mathrm{no} \mathrm{cell} \mathrm{in} Y\right)-\left(\mathrm{weighted} \mathrm{triplets} \mathrm{containing} \mathrm{no} \mathrm{cell} \mathrm{in} X\right)$$

is then normalized to ${\tilde{d}}_{\mathrm{WTCD}}\left(s\right)\in [-1,1]$ before it is written to the value table, similar to the procedures in TCD. For Player 2, the weighted evaluation is again $-d_{\mathrm{WTCD}}\left(s\right)$. This bias toward central control guides learning more quickly toward strong play.

Two-Step Look-Ahead

This heuristic evaluates both the current board and the tactical impact of each player’s next move. For every empty cell, we hypothetically place a marker for the acting player and count the resulting triplets. Let $d_{\mathrm{current}}\left(s\right)$ be the present TCD, or WTCD, at state s, and let $d_{\mathrm{next}}\left(s\right)$ be the maximal TCD, or WTCD, achievable on the next state of state s by either side. We define

$$d_{2\mathrm{SLA}}\left(s\right)=d_{\mathrm{current}}\left(s\right)+d_{\mathrm{next}}\left(s\right)$$

to be the two-step look-ahead (2SLA) heuristic and again rescale to $[-1,1]$ for our purpose. By incorporating short-term foresight, the agent prioritizes states that are most likely to yield additional triplets. It is straightforward to generalize this method to a k-step look-ahead (KSLA) scenario where dimensions are higher, but for this problem, it is not necessary.

These heuristics are illustrative rather than exhaustive. Any cheaply computed signal that correlates with long-run returns can serve as an evaluation function, provided that the extra overhead does not outweigh the savings in training time. Thus, the use of evaluation functions in RL can be intrinsically powerful depending on the application.

4. Results

We now turn from theoretical analysis to an empirical evaluation of our framework. Our experiments are designed to answer three key questions:

1.

Effect of evaluation functions. How much does seeding RL with domain-informed heuristics (in particular, TCD) accelerate convergence compared with a naive random initialization in the RL literature?

2.

Consistency with CGT. Once converged, do two frozen policies produce draws when played head-to-head, matching the CGT determinacy result?

3.

Evaluation against human opponents. How does the converged policy fare against human opponents, and does it confirm the theoretical first-player/second-player draw outcome in practice?

Table 2. A summary table highlighting how the triplet-coverage difference (TCD) evaluation function accelerates training without performance degradation, and how RL and human–machine matches confirm the theoretical draw outcome.

Experiment	Setup	Metric	Outcome
1. Effect of evaluation functions	One-step (temporal-difference) TD, ${ϵ}^{*}=0.05$, ${\alpha }^{*}=0.50$; win and draw rates tracked over a sliding window of $W=20,000$ episodes	Episodes to convergence (sample variances of cumulative win and draw rates less than $\delta ={10}^{-8}$)	Average random baseline: 190,087 episodes Average TCD heuristic: 146,262 episodes ($-23.1\%$)
2. Consistency with combinatorial game theory (CGT)	Frozen policies (no exploration); 100 head-to-head games	Fraction of draws	100% draws (matches CGT prediction of a draw under optimal play)
3. Evaluation against human opponents	Human participants vs. converged TCD policy (tested as Player 1 and Player 2)	Human win rate	0% human wins across all trials (aligns with RL optimal draw policy and CGT theory)

summarizes the results of all the experiments.

4.1. Execution Environment and Reproducibility

All the experiments were run on Microsoft Windows 11 Home (10.0.26100 Build 26100). The host CPU was an Intel Core Ultra 7 155H @ 1.40 GHz (16 cores, 22 threads) with 32 GB of RAM. The graphics acceleration (when used) came from an Intel Arc integrated GPU with approximately 2 GB of dedicated memory. The code was developed in Python 3.13.1, and the key libraries NumPy 2.2.1 and SciPy 1.16.1. Random seeds (121–130) were fixed to ensure reproducibility. Reproduction instructions and the exact code/data are provided in the repository cited in the Data Availability Statement.

4.2. Experiment 1: Effect of the Evaluation Functions

In Experiment 1, we measured how quickly TD converges when seeded with arbitrary initialization and TCD. To avoid bias, we first tuned the core RL hyperparameters under a zero-initialized baseline ($V^1\left(s\right)=V^2\left(s\right)=0$ for all nonterminal $s\in \mathcal{S}$) in Stage 1. These fixed hyperparameters were then used to compare random initialization against the TCD heuristic in Stage 2.

Stage 1: hyperparameter tuning under zero initialization

All training runs were executed as independent episodes of self-play. Each seed was run until the convergence criterion was met or until a hard upper bound of 1,000,000 episodes was reached. Convergence was declared when the empirical sample variances of the cumulative win and draw rates over a sliding window of length $W=20,000$ episodes fell below $\delta ={10}^{-8}$. Mathematically speaking, define the estimated cumulative Player 1 win rate after n episodes by

$${\widehat{p}}_1\left(n\right)={\displaystyle \frac{\# \mathrm{Player} 1 \mathrm{wins} \mathrm{up} \mathrm{to} \mathrm{episode} n}{n}},$$

and similarly, the estimated cumulative draw rate by

$${\widehat{p}}_d\left(n\right)={\displaystyle \frac{\# \mathrm{draws} \mathrm{up} \mathrm{to} \mathrm{episode} n}{n}}.$$

After every episode, we appended the tuple $({\widehat{p}}_1\left(n\right),{\widehat{p}}_d\left(n\right))$ to a history buffer. When $n\ge W$ episodes had elapsed, we extracted the last W entries and formed the two sequences of length W corresponding to ${\widehat{p}}_1$ and ${\widehat{p}}_d$. We then computed the empirical sample variances of these two sequences, denoted $s_{1,W}^2$ and $s_{d,W}^2$. Convergence was declared at the first episode for which both sample variances were below the threshold $\delta$ (i.e., $s_{1,W}^2<\delta$ and $s_{d,W}^2<\delta$). Note that Player 2’s cumulative win rate is implicit since ${\widehat{p}}_2\left(n\right)=1-{\widehat{p}}_1\left(n\right)-{\widehat{p}}_d\left(n\right)$, and therefore monitoring ${\widehat{p}}_1$ and ${\widehat{p}}_d$ suffices to detect stabilization of all three outcome proportions.

This procedure ensures that stabilization is required for both players before termination. Such a tuning setup ensures that any observed speed-ups in Stage 2 stem from initialization rather than hyperparameter differences. We performed a grid search over

$$ϵ\in \{0.05,0.10,0.20,0.30\},\alpha \in \{0.05,0.10,0.20,0.50\},$$

running five independent seeds per pair (seeds 121–125). For each pair, we recorded the per-seed episode count at which the convergence criterion was first met, then computed the sample mean and sample standard deviation across the five seeds. We selected $({ϵ}^{*},{\alpha }^{*})=(0.05,0.50)$ because it attained one of the smallest mean convergence times while maintaining acceptably low variability.

Table 3. Sample mean (± sample standard deviation) number of episodes to convergence under zero initialization for each $(ϵ,\alpha )$ pair (five independent seeds). Convergence is declared when the sample variances of cumulative win and draw rates over a 20,000-episode sliding window fall below ${10}^{-8}$. All values are rounded to the nearest integer.

$\mathit{ϵ}\setminus \mathit{\alpha }$	0.05	0.10	0.20	0.50
0.05	259,093 (±70,243)	244,966 (±98,232)	194,820 (±39,686)	177,848 (±19,414)
0.10	349,362 (±171,793)	337,401 (±112,093)	219,956 (±36,977)	188,379 (±21,125)
0.20	637,247 (±131,163)	359,004 (±54,332)	188,616 (±18,676)	205,076 (±32,437)
0.30	719,999 (±98,152)	343,408 (±40,623)	202,829 (±23,446)	250,091 (±34,223)

provides these results, and we present the raw seed-by-seed $(ϵ,\alpha )$ tuple data in

Table A1. Raw episodes-to-convergence for each $(ϵ,\alpha )$ pair and each random seed (Seeds 121–125) using the zero initialization baseline. Convergence is defined by the sample variances of the cumulative win and draw rates being both less than $\delta ={10}^{-8}$ over a $W=20,000$-episode sliding window.

$\mathit{ϵ}$	$\mathit{\alpha }$	Seed 121	Seed 122	Seed 123	Seed 124	Seed 125
0.05	0.05	310,033	257,279	145,606	258,198	324,347
	0.10	387,648	154,423	300,944	167,846	213,967
	0.20	205,384	128,614	214,257	193,277	232,569
	0.50	178,748	160,856	210,678	169,766	169,192
0.10	0.05	219,702	310,615	371,382	212,151	632,958
	0.10	481,303	234,146	388,509	213,843	369,202
	0.20	245,052	220,414	262,556	167,146	204,613
	0.50	196,265	205,237	177,817	206,053	156,525
0.20	0.05	782,386	579,277	476,076	585,719	762,779
	0.10	351,709	377,120	334,538	439,090	292,565
	0.20	180,459	204,137	206,206	160,858	191,421
	0.50	170,843	212,443	171,355	233,151	237,589
0.30	0.05	731,808	562,911	711,081	767,505	826,691
	0.10	337,431	319,809	295,909	362,175	401,717
	0.20	237,782	186,920	190,416	182,962	216,065
	0.50	261,807	216,197	211,378	285,348	275,726

. All subsequent comparisons use $({ϵ}^{*},{\alpha }^{*})$.

Stage 2: initialization comparison (random vs. TCD)

With ${ϵ}^{*}=0.05$ and ${\alpha }^{*}=0.50$ fixed, we compared two initialization schemes:

• Random initialization (baseline): For all nonterminal $s\in \mathcal{S}$,

  $$V^1\left(s\right),V^2\left(s\right)\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }Uniform(-1,1).$$
• TCD heuristic: For each nonterminal state s, we computed the normalized TCD ${\tilde{d}}_{\mathrm{TCD}}\left(s\right)\in [-1,1]$ for the player-to-move. We then initialized the tables by setting

  $$V^1\left(s\right)\leftarrow \left\{\begin{array}{cc}{\tilde{d}}_{\mathrm{TCD}}\left(s\right),\hfill & \mathrm{if}\mathrm{Player}1\mathrm{is}\mathrm{to}\mathrm{move}\mathrm{in}s,\hfill \\ -{\tilde{d}}_{\mathrm{TCD}}\left(s\right),\hfill & \mathrm{if}\mathrm{Player}2\mathrm{is}\mathrm{to}\mathrm{move}\mathrm{in}s,\hfill \end{array}\right.V^2\left(s\right)\leftarrow -V^1\left(s\right).$$

For each initialization scheme (random vs. TCD), we ran five new independent seed runs (seeds 126–130) and recorded the episodes required to reach the convergence criterion (similarly, sample variances less than $\delta ={10}^{-8}$ for both the cumulative win and draw rates over a sliding window $W=20,000$). The random initialization group produced sample mean episodes to convergence 190,087 with a sample standard deviation 24,216, while the TCD group produced a sample mean 146,262 with a sample standard deviation 32,020. The results are recorded in

Table 4. Sample mean and sample standard deviation number of episodes to convergence under ${ϵ}^{*}=0.05$, ${\alpha }^{*}=0.50$, comparing random initialization vs. the TCD heuristic (five new independent seeds). Convergence is declared when the sample variances of cumulative win and draw rates over a 20,000-episode window fall below ${10}^{-8}$. All values are rounded to the nearest integer except for the percentage reduction.

Initialization	Mean Episodes	Standard Deviation	Reduction vs. Random
Random initialization (baseline)	190,087	24,216
TCD heuristic	146,262	32,020	≈$-23.1$%

, and raw seed-by-seed data are presented in

Table A2. Raw episodes-to-convergence for the comparison between the random baseline and TCD under the selected hyperparameters ${ϵ}^{*}=0.05$ and ${\alpha }^{*}=0.50$ for each random seed (Seeds 126–130). Convergence is defined by the sample variances of the cumulative win and draw rates being both less than $\delta ={10}^{-8}$ over a $W=20,000$-episode sliding window.

Initialization	Seed 126	Seed 127	Seed 128	Seed 129	Seed 130
Random Baseline	224,547	200,676	162,123	189,229	173,859
TCD Heuristic	195,185	149,118	112,231	123,535	151,241

.

Next, we verified key assumptions required and conducted a standard two-sample (Student’s) t test on the per-seed convergence times to establish statistical significance:

• Independence check: By design, each seed run is an independent experiment, so the two groups (random vs. TCD) are independent.
• Normality checks (Shapiro–Wilk test): The Shapiro–Wilk test [78] tests the null hypothesis that the data are from a normal distribution. We have p-value = 0.9382 for the random initialization sample and p-value = 0.6096 for the TCD sample. We fail to reject the normality assumption for both samples at the 5% level of significance.
• Equal-variance check (Levene’s test): Levene’s test [79] tests the null hypothesis that all input samples are from populations with equal variances. We have p-value = 0.7088. We fail to reject the homoscedasticity assumption at the 5% level.
• Two-sample t test: After confirming the above assumptions, we can proceed with a standard t test. A two-sample t test [80] is a test for the null hypothesis that two independent samples have identical means, assuming that the populations have identical variances. We have an upper one-sided p-value = 0.0203. We reject the null that the two population means are the same, in favor of a significant reduction at the level of 5%.

Practically, this corresponds to an average reduction of 23.1% in episodes to convergence, which translates to substantial computational savings in repeated training scenarios. We expect WTCD and 2SLA heuristics to improve efficiency further, although their computational cost must be weighed against this benefit.

To obtain explicit policies from the converged estimates, we froze the learned value tables $V^1$ and $V^2$ and derived deterministic greedy policies ${\pi }_{*}^1$ and ${\pi }_{*}^2$. Concretely, for each reachable nonterminal state s, we first determined the player to move and then selected the greedy action according to that player’s table:

$${\pi }_{*}^1\left(s\right)=\underset{a\in \mathcal{A}\left(s\right)}{arg\; max}{V}^1(Update(s,a))\mathrm{and}{\pi }_{*}^2\left(s\right)=\underset{a\in \mathcal{A}\left(s\right)}{arg\; max}{V}^2(Update(s,a)),$$

where only the relevant policy ${\pi }_{*}^i$ is consulted when player i is to move in s. Ties among maximizers were resolved randomly by selecting any action leading to the highest value.

4.3. Experiment 2: Consistency with Combinatorial Game Theory

The second experiment tests whether the frozen deterministic greedy policies yield 100% draws in head-to-head play. After convergence for each random seed between 126 and 130, for each heuristic, we froze the learned policies with no exploration ($ϵ=0$) and played 100 head-to-head matches between two frozen converged policies ${\pi }_{*}^1$ and ${\pi }_{*}^2$. All 100 plays ended in draws, in exact agreement with the CGT determinacy result.

4.4. Experiment 3: Evaluation Against Human Opponents

The third experiment evaluates the learned policy against human participants in both roles. We invited volunteers to play as Player 1 against the converged TCD policy. In every trial, the human participants failed to win. Repeating the experiment with participants assigned to Player 2 yielded the same outcome (0% win rate), further corroborating that the learned policy attains the game-theoretic draw.

5. Discussion

Building on the canonical $3\times 3$ tic-tac-toe, we develop a novel cumulative tic-tac-toe framework and analyze it through the complementary lenses of CGT and RL. We first formalize the variant and outline six real-world scenarios, and then establish its key theoretical properties within the CGT framework. Next, we cast the game as an MDP and validate optimal play via one-step TD learning. To accelerate learning, we seeded the estimated state-value function with the domain-informed TCD heuristic, yielding a statistically significant 23.1% reduction in convergence episodes relative to a random initialization while preserving the draw outcome predicted by CGT in our head-to-head and human tests. Convergence was declared when the sample variances of cumulative win and draw rates fell below $\delta ={10}^{-8}$ over a $W=20,000$-episode sliding window, ensuring genuine stability rather than transient fluctuations.

We emphasize that our operational convergence criterion signals empirical stabilization of the training process across episodes and seeds. It does not constitute a formal mathematical proof of global optimality. In other words, the criterion indicates that the policy is stable under the chosen training regimen, and, in our tests, the stabilized policies produced the theorized draw outcome, but a mathematical guarantee of optimality would require additional analysis (e.g., proving monotone improvement and eventual convergence to the minimax value under the given learning rates and exploration schedule).

The present study employs a full tabular representation of the state-value function. Scaling to larger boards will require function-approximation techniques (e.g., linear architectures or deep neural networks) and more systematic hyperparameter optimization. All hyperparameter values in our prototype were chosen empirically since no formal guidelines exist at this stage. Our limited sweep over 16 pilot combinations (each averaged over five random seeds) yielded one representative setting $({ϵ}^{*},{\alpha }^{*})$. A more comprehensive grid search or Bayesian optimization on all hyperparameters, evaluated over five to ten seeds per setting, is planned to substantiate and stabilize these choices. In future work, we will investigate WTCD and 2SLA to determine if their computational overhead is outweighed by further speed-ups. We believe that incorporating domain-informed evaluation functions is a promising direction in RL, with potential for substantial improvements in training efficiency and stability.

Our combined CGT–RL approach could also be extended to multiplayer, stochastic, or imperfect-information games. These generalizations introduce new challenges (e.g., uncertainty and hidden information), but provide richer, more realistic testbeds for future study. Future work will also tailor the cumulative tic-tac-toe framework to specific application domains by adapting the board, action set, and scoring rules. For example, we plan to apply the model to strategic resource allocation and multi-agent planning to assess how well cumulative tic-tac-toe captures domain-specific features and to establish benchmark problems for CGT and RL research.

Last but not least, we used tabular one-step TD as a proof-of-concept in this study. Alternative RL approaches, such as Q-learning, MC control, actor–critic methods, and deep neural-network-based RL, have different empirical trade-offs in convergence speed, sample efficiency, and generalization. A systematic benchmark should compare these algorithms on cumulative tic-tac-toe, with and without CGT-informed evaluation initializations, using metrics such as episodes-to-convergence, stability across seeds, and computational cost (wall-clock time and per-episode complexity). Related evaluations in other decision problems provide guidance and suggest that algorithm choice interacts nontrivially with problem structure and initialization.