Constructing Non-Markovian Decision Process via History Aggregator

Abstract

In the domain of algorithmic decision-making, non-Markovian dynamics manifest as a significant impediment, especially for paradigms such as Reinforcement Learning (RL), thereby exerting far-reaching consequences on the advancement and effectiveness of the associated systems. Nevertheless, the existing benchmarks are deficient in comprehensively assessing the capacity of decision algorithms to handle non-Markovian dynamics. To address this deficiency, we have devised a generalized methodology grounded in category theory. Notably, we established the category of Markov Decision Processes (MDP) and the category of non-Markovian Decision Processes (NMDP), and proved the equivalence relationship between them. This theoretical foundation provides a novel perspective for understanding and addressing non-Markovian dynamics. We further introduced non-Markovianity into decision-making problem settings via the History Aggregator for State (HAS). With HAS, we can precisely control the state dependency structure of decision-making problems in the time series. Our analysis demonstrates the effectiveness of our method in representing a broad range of non-Markovian dynamics. This approach facilitates a more rigorous and flexible evaluation of decision algorithms by testing them in problem settings where non-Markovian dynamics are explicitly constructed.

1. Introduction

Decision-making algorithms, such as RL, generally demand that the problem satisfy the Markov assumption, which posits that the current observation encapsulates all accessible information for decision-making, rendering past observations irrelevant. However, many real-world systems do not adhere to this assumption. For example, control systems often exhibit non-Markovianity due to limited sensor information about internal states [1]. Similarly, open quantum systems display non-Markovian characteristics due to complex interactions with their external environment [2]. Such non-Markovian environments are prevalent in real-world applications, including human physiology, biological systems, and material science [3].

To characterize and solve non-Markovian decision problems, a variety of theoretical models beyond the basic MDP formulation have been proposed. These approaches primarily follow two conceptual strands: state abstraction and reward specification.

The first strand aims to convert an NMDP into a solvable MDP by learning a compact representation of history. Markov Abstraction methods [4,5,6,7,8,9,10] fall into this category, where the core idea is to compress the entire history or learn an approximate Markovian state, effectively “solving” the non-Markovianity through representation learning. While powerful, the quality of the abstraction is algorithm-dependent and may inadvertently alter the problem’s inherent difficulty due to information loss.

The second strand focuses on formally specifying the non-Markovian structure, particularly in reward functions. Automaton or logic-based models, such as Reward Machines (RM) [11,12] and Regular Decision Processes (RDP) [13,14], use finite-state automata or temporal logic to explicitly encode complex task objectives and reward dependencies over time. These methods excel at providing interpretable task specifications but are primarily designed for non-Markovian rewards rather than general state transition dynamics.

While offering valuable tools, these existing theoretical models are diverse in formulation and objectives. Crucially, they are often designed to eliminate or specify non-Markovianity for the purpose of solving a given problem. In contrast, there lacks a unified framework to systematically construct non-Markovian problems with controllable temporal dependencies from first principles, a capability essential for generating standardized benchmarks to impartially evaluate algorithm performance. This gap motivates our work.

Table 1 compares our approach with representative paradigms in the literature at the conceptual level. Unlike methods that aim to “solve” non-Markovianity (e.g., by learning abstractions, specifying rewards, or handling partial observability), our HAS framework aims to “construct” non-Markovianity. This provides a generative paradigm that starts from a base MDP and yields NMDPs with clear, controllable dependency structures. Consequently, it enables a pure evaluation of an algorithm’s capability in history memorization and decoding, disentangled from changes in the inherent problem difficulty or confusion with distinct problem classes like POMDPs.

Table 1. Comparison with representative paradigms for handling non-Markovianity.

Paradigm/Method	Our Work (HAS-Based Construction)	State Abstraction/Representation Learning	Automaton/Logic-Based Methods
Core Philosophy & Goal	Systematic construction: Create NMDPs with precisely controllable temporal dependencies from known MDPs, for algorithm evaluation.	Compressed solving: Learn a Markovian state representation from history to simplify solving the original NMDP.	Task specification: Use automata or temporal logic to formally define non-Markovian reward functions or task objectives.
Source of Non-Markovianity	Injected externally via reversible history aggregators applied to the state sequence; dependency is structured and known.	Arises from the learning process of finding a compressed representation of history; dependency is implicit and data-driven.	Arises from explicit logical encoding of the reward generation mechanism or task constraints (e.g., reward machines, LTL formulas).
Key Properties	1. Precise and flexibly controllable temporal dependency structure. 2. Preserves problem essence (optimal performance upper bound unchanged under reversibility). 3. General and reproducible benchmark generation framework.	1. Aims to discover effective representations, not control dependencies. 2. May alter problem difficulty due to information loss or bias. 3. Representation quality depends on specific learning algorithms.	1. Excels at modeling structured rewards, not general non-Markovian state transitions. 2. Provides clear semantics for task understanding and decomposition.

In response, we establish the MDP category and the NMDP category, providing a unified and rigorous view within the framework of category theory. Under this framework, any general approach that converts NMDPs into MDPs while preserving the transition dynamics can be seen as a functor from the NMDP category to the MDP category. This framework not only encapsulates the essence of preceding research but also paves the way for devising novel strategies to tackle non-Markovian decision problems. Moreover, the utilization of the category theory perspective facilitates a deeper understanding of the relationship between MDP and NMDP. We prove that the MDP category and the NMDP category are equivalent categories based on the intuition that any NMDP can be transformed into an MDP, given that the entire history can be regarded as a state.

Numerous algorithms have emerged to solve non-Markovian decision problems. However, an algorithm that is both commonly accepted and highly effective for NMDPs remains elusive. The evaluation of such algorithms relies on non-Markovian decision problems, which poses a set of challenges. For instance, ref. [15] utilizes a multi-armed bandit and robot grid world with non-Markovian rewards; ref. [3] employs a blood glucose control simulation environment; ref. [5] uses Rotating MAB, Malfunction MAB, Enemy Corridor, Reset—Rotating MAB, and Flickering grid; ref. [16] uses an office grid-world scenario; ref. [17] uses modified Mujoco environments; ref. [18] uses modified Cartpole and mountain Car, along with a controlled non-Markovian random walk. The non-Markovian environments described in the literature, which are scattered and unsystematic, originate from diverse fields. Moreover, most of these environments are either obtained through simple construction or derived from masking some part of the observation in existing MDP, that is, by transforming it into a Partially Observable MDP (POMDP), which may change the internal difficulty because of information loss in the observation. Their simplicity and lack of standardization impede the development of a universal benchmark, thereby limiting the ability to comprehensively evaluate how effectively algorithms can manage non-Markovian dynamics.

To address this gap, we develop a method for constructing NMDPs from MDPs via a reversible History Aggregator for State (HAS). The core idea is to inject controllable temporal dependencies into the state representation by aggregating historical states. The reversibility condition is crucial: it ensures the original MDP state can be losslessly decoded from the NMDP’s full history, thereby preserving the problem’s inherent transition dynamics and optimal performance bound. This allows for a pure assessment of a decision-making algorithm’s core capability in memorizing and decoding history to manage non-Markovianity, isolated from changes in base problem difficulty.

We propose two practical, reversible HAS implementations. The first employs a group operator (e.g., vector addition) for aggregation, inspired by how prefix sums increase Markov order. The second uses convolution with an auxiliary sequence to assign time-decaying weights to past states. As states are typically real-valued vectors, both methods are efficiently implemented using linear algebra. Critically, the constructed NMDPs maintain the same interface as the original MDP, enabling the direct application of any algorithm without modification. This facilitates a direct comparison of performance degradation solely attributable to introduced non-Markovian dependencies.

In summary, this work establishes a new paradigm for principled construction and evaluation in non-Markovian decision-making. Our primary contributions are:

• A Unifying Categorical Foundation: We establish and prove the equivalence of the MDP and NMDP categories, providing a unified theoretical lens.
• A Generative Framework for Controlled Non-Markovianity: We introduce the reversible HAS framework to systematically construct NMDPs with precise, tunable temporal dependencies from any base MDP.
• Practical, Tunable Construction Methods: We instantiate the framework with two implementable HAS classes (group-based and convolution-based), enabling fine-grained control over dependency structures.
• A Practical Benchmarking Tool: We provide efficient Gymnasium-compatible wrappers to seamlessly convert standard RL benchmarks, enabling reproducible, pure evaluation of algorithmic memory and decoding capabilities.

The ultimate goal is to shift from ad-hoc problem adaptation to principled problem construction. The outcome is a coherent pipeline that transforms canonical MDPs into a family of NMDP benchmarks with clear, tunable non-Markovian characteristics, filling a critical gap in the evaluation toolbox for sequential decision-making.

In Section 2, we introduce the fundamentals of category theory, MDP, and NMDP, along with the basics of algebra. This serves as essential background for understanding our work. In Section 3, we formally describe and prove the equivalence of the MDP and NMDP categories. This not only unifies the two concepts but also naturally leads to the method of constructing NMDPs in the subsequent section. Section 4 presents the definition of HAS, two construction methods using reversible HAS, and an analysis of their properties. These methods innovatively introduce non-Markovianity while preserving the original nature of the problem. Although the main contribution of our work is theoretical, we convert some classical MDPs into NMDPs and conduct several empirical experiments using some RL algorithms in Section 5. This validates our theoretical framework and demonstrates its applicability. The Appendix A, Appendix B, Appendix C, Appendix D, Appendix E, Appendix F, Appendix G, Appendix H, Appendix I, Appendix J, Appendix K and Appendix L provide mathematical proofs, illustrative examples, and intuitive explanations of key concepts. This allows the main text to focus on the key ideas without being cluttered with details. Overall, our work offers a comprehensive, novel, and practical solution for constructing non-Markovian decision problems.

2. Preliminaries

2.1. Notations

In the following, we denote by ${\Delta }_X:=\{p\mid p:X\to [0,\infty ),{\sum }_{x\in X}p\left(x\right)=1\}$ the set of all probability distributions on the set X, denote by $f^{⟨n⟩}$ an n-variable function created from the unary function f in the manner of $f^{⟨n⟩}(x_1,x_2,\dots ,x_n):=(f\left(x_1\right),f\left(x_2\right),\dots ,f\left(x_n\right))$, and denote by $x_{m:n}(m,n\in \mathbb{N},m\le n)$ a finite sequence ${\left\{x_i\right\}}_{i=m}^n$. We do not distinguish between $x_{n:n}$ and $x_n$ in the following.

2.2. Category Theory

A category is a system of related objects. The objects do not live in isolation: there is some notion of morphism, or equivalently, map, between objects, binding them together [19]. Category theory delves into the relationships between objects and morphisms. Objects can be of any nature, and morphisms denote the relations or transformations linking these objects. For example, all groups together with all group homomorphisms form a category of groups, and any partially ordered set can also form a category. This theory offers a potent framework for apprehending mathematical structures. By distilling common traits, category theory allows for a unified analysis across diverse mathematical domains. It uncovers profound connections and streamlines intricate concepts, presenting a more coherent view of mathematics.

Definition 1 (Category).

A category $\mathfrak{C}$ consists of:

• a collection $ob\left(\mathfrak{C}\right)$ of  objects;
• for each $\mathcal{A},\mathcal{B}\in ob\left(\mathfrak{C}\right)$, a collection $\mathfrak{C}(\mathcal{A},\mathcal{B})$ of  morphisms  from $\mathcal{A}$ to $\mathcal{B}$;
• for each $\mathcal{A},\mathcal{B},\mathcal{C}\in ob\left(\mathfrak{C}\right)$, a function

  $$\begin{array}{ccc}\mathfrak{C}(\mathcal{A},\mathcal{B})\times \mathfrak{C}(\mathcal{B},\mathcal{C})& \to & \mathfrak{C}(\mathcal{A},\mathcal{C})\hfill \\ (g,f)& \mapsto & g\circ f\hfill \end{array}$$
  called  composition;
• for each $\mathcal{A}\in ob\left(\mathfrak{C}\right)$, an element $1_{\mathcal{A}}\in \mathfrak{C}(\mathcal{A},\mathcal{A})$, called the  identity  on $\mathcal{A}$.

satisfying the following axioms:

• associativity: for each $f\in \mathfrak{C}(\mathcal{A},\mathcal{B})$, $g\in \mathfrak{C}(\mathcal{B},\mathcal{C})$, $h\in \mathfrak{C}(\mathcal{C},\mathcal{D})$ we have $(h\circ g)\circ f=h\circ$$(g\circ f)$;
• identity: for each $f\in \mathfrak{C}(\mathcal{A},\mathcal{B})$, we have $f=f\circ 1_{\mathcal{A}}=1_{\mathcal{B}}\circ f$.

Just as morphisms convey intra-category object relations or transformations, functors represent inter-category ones.

Definition 2 (Functor).

Let $\mathfrak{A},\mathfrak{B}$ be categories. A functor $\mathbf{F}:\mathfrak{A}\to \mathfrak{B}$ consists of:

• a function $ob\left(\mathfrak{A}\right)\to ob\left(\mathfrak{B}\right)$ written as $\mathcal{A}\mapsto \mathbf{F}\left(\mathcal{A}\right)$;
• for each $\mathcal{A},{\mathcal{A}}^{\prime }\in \mathfrak{A}$, a function

  $$\mathfrak{A}(\mathcal{A},{\mathcal{A}}^{\prime })\to \mathfrak{B}(\mathbf{F}\left(\mathcal{A}\right),\mathbf{F}\left({\mathcal{A}}^{\prime }\right))$$
  written as $f\mapsto \mathbf{F}\left(f\right)$; satisfying the following axioms:

  –

  $\mathbf{F}(f^{\prime }\circ f)=\mathbf{F}\left(f^{\prime }\right)\circ \mathbf{F}\left(f\right)$ if $\mathcal{A}\stackrel{f}{\to }{\mathcal{A}}^{\prime }\stackrel{f^{\prime }}{\to }{\mathcal{A}}^{\prime \prime }$ in $\mathfrak{A}$;

  –

  $\mathbf{F}\left(1_{\mathcal{A}}\right)=1_{\mathbf{F}\left(\mathcal{A}\right)}$ if $\mathcal{A}\in \mathfrak{A}$.

The collection of functors from $\mathfrak{A}$ to $\mathfrak{B}$ also forms a category $\mathbf{Fun}(\mathfrak{A},\mathfrak{B})$.

2.3. Algebra

Definition 3 (Semigroup).

Let S be a non-empty set, $·:S\times S\to S$ is a binary operator, if the following axioms hold:

• closure: $\forall a,b\in S,a·b\in S$;
• associativity: $\forall a,b,c\in S,a·(b·c)=(a·b)·c$.

Then $(S,·)$ forms a semigroup.

If there exists a unit $e\in S$ in a semigroup $(S,·)$ such that $\forall a\in S,a·e=e·a=a$, then $(S,·)$ becomes a monoid.

Definition 4 (Group).

Let $(G,·)$ be a monoid, e is the unit, if the following axiom holds:

• inverse: $\forall a\in G,\exists a^{-1}\in G,a^{-1}·a=a·a^{-1}=e$.

Then $(G,·)$ forms a group.

If the operator in a group $(G,·)$ is commutative, i.e., $\forall a,b\in G,a·b=b·a$, then $(G,·)$ is an Abelian group.

Definition 5 (Ring).

Let R be a non-empty set, if $(R,·)$ forms a semigroup, $(R,+)$ forms an Abelian group with unit $0\in R$, and the following axioms hold:

• left distributivity: $\forall a,b,c\in R,a·(b+c)=a·b+a·c$;
• right distributivity: $\forall a,b,c\in R,(b+c)·a=b·a+c·a$.

Then $(R,+,·)$ forms a ring.

If the operator · in ring $(R,+,·)$ is commutative, then $(R,+,·)$ becomes a commutative ring, and if $(R,·)$ is a monoid with unit 1, then $(R,+,·)$ becomes a unital ring with unit 1.

Definition 6 (Left $\mathit{R}$-Module).

Let $(R,+,·)$ be a ring, $(M,\oplus )$ be an Abelian group, if there exists a map $R\times M\to M$ under which the image of any $(r,m)\in R\times M$ is denoted as $rm\in M$, and the following axioms hold:

• $\mathit{R}$-distributivity:

  $$\forall r\in R,\forall m,m^{\prime }\in M,r(m\oplus m^{\prime })=rm\oplus r{m}^{\prime }$$
• $\mathit{M}$-distributivity:

  $$\forall r,r^{\prime }\in R,\forall m\in M,(r+r^{\prime })m=rm\oplus r^{\prime }m$$
• associativity:

  $$\forall r,r^{\prime }\in R,\forall m\in M,r\left(r^{\prime }m\right)=(r·r^{\prime })m$$
• identity: if $(R,+,·)$ is a unital ring with unit 1, then $\forall m\in M,1m=m$.

Then M is called a left R-module.

A left R-module is a generalization of a linear space. When R becomes a field, left R-module M becomes a linear space.

2.4. MDP and NMDP

Before setting up the MDP and NMDP categories, we define them.

Definition 7 (Markov Decision Process (MDP)).

An MDP $\mathcal{M}$ is a tuple $⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ consisting of:

• Initial state distribution ${\rho }_0\in {\Delta }_S$;
• State set S;
• Action set A;
• Transition dynamics at time t:

  $$T_t:S\times A\to {\Delta }_{S\times \mathbb{R}}$$

A policy for an MDP at time t is defined as ${\pi }_t:S\to {\Delta }_A$. An MDP is called degenerate if $\exists s,s^{\prime }\in S,\forall t\in \mathbb{N},\forall a\in A,T_t(s,a)=T_t(s^{\prime },a)$, which means there are indistinguishable states in S.

Note that, by definition, an MDP is not necessarily time-homogeneous. However, any non-time-homogeneous MDP can be converted into a time-homogeneous MDP by extending the state set S to be a subset of $\mathbb{N}\times S$, where each state is associated with its timestep, i.e., $s_t\mapsto (t,s_t)$ and $T:((t,s_t),a)\mapsto T_t(s_t,a)$.

Definition 8 (Non-Markov Decision Process (NMDP)).

An NMDP $\mathcal{N}$ is a tuple $⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ consisting of:

• Initial state distribution ${\rho }_0\in {\Delta }_S$;
• State set S;
• Action set A;
• Transition dynamics at time t:

  $$T_t:S^{t+1}\times A^t\times {\mathbb{R}}^t\times A\to {\Delta }_{S\times \mathbb{R}}$$

A policy for an NMDP at time t is ${\pi }_t:S^{t+1}\times A^t\times {\mathbb{R}}^t\to {\Delta }_A$. The history set at time $\mathbf{t}$ for an MDP or an NMDP is $H_t:=S^{t+1}\times A^t\times {\mathbb{R}}^t$, with which transition dynamics and policy of an NMDP can be rewritten as $T_t:H_t\times A\to {\Delta }_{S\times \mathbb{R}}$, ${\pi }_t:H_t\to {\Delta }_A$. The history set H for an MDP or an NMDP is defined as the disjoint union of all $H_t$, i.e., $H:={⨆}_{t=0}^{\infty }{H}_t$. The proper prefix relation between h and $h^{\prime }$ is denoted as $h\prec h^{\prime }$, indicating that h is a proper prefix of $h^{\prime }$, i.e., $h=(s_{0:t},a_{0:t-1},r_{0:t-1})$ and $h^{\prime }=(s_{0:t+k},a_{0:t+k-1},r_{0:t+k-1}),k\in {\mathbb{N}}^{+}$.

With the concept of the history set for both MDPs and NMDPs, we can define operators that extract states, actions, and rewards from a history.

Definition 9 (States, Actions, and Rewards Extraction Operators).

The states, actions, and rewards extraction operators are:

• $E_S:h_t\mapsto s_{0:t}(\forall t\in \mathbb{N},\forall h_t\in H_t)$;
• $E_A:h_t\mapsto a_{0:t-1}(\forall t\in {\mathbb{N}}^{+},\forall h_t\in H_t)$;
• $E_{\mathbb{R}}:h_t\mapsto r_{0:t-1}(\forall t\in {\mathbb{N}}^{+},\forall h_t\in H_t)$.

Definition 10 (Latest State, Action, and Reward Extraction Operators).

The latest state, action, and reward extraction operators at time t are:

• $L_{S,t}:h_t\mapsto s_t(\forall t\in \mathbb{N},\forall h_t\in H_t)$;
• $L_{A,t}:h_{t+1}\mapsto a_t(\forall t\in \mathbb{N},\forall h_{t+1}\in H_{t+1})$;
• $L_{\mathbb{R},t}:h_{t+1}\mapsto r_t(\forall t\in \mathbb{N},\forall h_{t+1}\in H_{t+1})$.

3. Equivalence of MDP and NMDP Category

In this section, we present the definitions of the MDP category and the NMDP category, together with the theorem regarding their equivalence relationship. The proofs of their well-definedness are provided in the Appendix A and Appendix B.

Definition 11 (MDP Category).

$\mathfrak{M}$ is a category where:

• Objects: The objects are MDPs in the form of $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$;
• Morphisms: A morphism $\varphi =({\varphi }_S,{\varphi }_A,{\varphi }_{\mathbb{R}})$ from $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ to ${\mathcal{M}}^{\prime }=⟨{\rho }_0^{\prime },S^{\prime },A^{\prime },{\left\{T_t^{\prime }\right\}}_{t=0}^{\infty }⟩$ where ${\varphi }_S:S\to S^{\prime }$, ${\varphi }_A:A\to A^{\prime }$, ${\varphi }_{\mathbb{R}}:\mathbb{R}\to \mathbb{R}$, satisfies the following properties:

  –

  ${\rho }_0={\rho }_0^{\prime }\circ {\varphi }_S$;

  –

  $T_t(s,a)=\left((T_t^{\prime }\circ ({\varphi }_S,{\varphi }_A))(s,a)\right)\circ ({\varphi }_S,{\varphi }_{\mathbb{R}})$, $(\forall t\in \mathbb{N},\forall s\in S,\forall a\in A)$.

  Identity: $1_{\mathcal{M}}=(1_S,1_A,1_{\mathbb{R}})$;Composition: composition of $({\varphi }_S^{\prime },{\varphi }_A^{\prime },{\varphi }_{\mathbb{R}}^{\prime })$ and $({\varphi }_S,{\varphi }_A,{\varphi }_{\mathbb{R}})$ is $({\varphi }_S^{\prime }\circ {\varphi }_S,{\varphi }_A^{\prime }\circ {\varphi }_A,{\varphi }_{\mathbb{R}}^{\prime }\circ {\varphi }_{\mathbb{R}})$.

Definition 12 (NMDP Category).

$\mathfrak{N}$ is a category where:

• Objects: The objects are NMDPs in the form of $\mathcal{N}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$;
• Morphisms: A morphism $\psi =({\psi }_S,{\psi }_A,{\psi }_{\mathbb{R}})$ from $\mathcal{N}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ to ${\mathcal{N}}^{\prime }=⟨{\rho }_0^{\prime },S^{\prime },A^{\prime },{\left\{T_t^{\prime }\right\}}_{t=0}^{\infty }⟩$ where ${\psi }_S:S\to S^{\prime }$, ${\psi }_A:A\to A^{\prime }$, ${\psi }_{\mathbb{R}}:\mathbb{R}\to \mathbb{R}$, satisfies the following properties:

  –

  ${\rho }_0={\rho }_0^{\prime }\circ {\psi }_S$;

  –

  $T_t(h_t,a)=\left((T_t^{\prime }\circ ({\psi }_{H_t},{\psi }_A))(h_t,a)\right)\circ ({\psi }_S,{\psi }_{\mathbb{R}})$, $(\forall t\in {\mathbb{N}}^{+},\forall h_t\in H_t,\forall a\in A)$, where ${\psi }_{H_t}=({\psi }_S^{⟨t+1⟩},{\psi }_A^{⟨t⟩},{\psi }_{\mathbb{R}}^{⟨t⟩})$.

  Identity: $1_{\mathcal{N}}=(1_S,1_A,1_{\mathbb{R}})$;  Composition: composition of $({\psi }_S^{\prime },{\psi }_A^{\prime },{\psi }_{\mathbb{R}}^{\prime })$ and $({\psi }_S,{\psi }_A,{\psi }_{\mathbb{R}})$ is $({\psi }_S^{\prime }\circ {\psi }_S,{\psi }_A^{\prime }\circ {\psi }_A,{\psi }_{\mathbb{R}}^{\prime }\circ {\psi }_{\mathbb{R}})$.

A morphism in either category represents that the transition dynamics of the source decision process can be “simulated” by the target.

As discussed in Section 1, previous algorithmic works have sought functors in $\mathbf{Fun}(\mathfrak{N},\mathfrak{M})$, which can generally be summarized as approximations of the Markov abstraction functor $\mathbf{M}\in \mathbf{Fun}(\mathfrak{N},\mathfrak{M})$, utilizing various techniques to compress history. In contrast to what is described in the existing literature, our definition of Markov abstraction retains the entire history without any compression.

Definition 13 (Markov Abstraction).

Markov abstraction is a functor $\mathbf{M}:\mathfrak{N}\to \mathfrak{M}$. For any $\mathcal{N}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩\in \mathfrak{N}$ there exists ${\mathcal{M}}^{\prime }=\mathbf{M}\left(\mathcal{N}\right)=⟨{\rho }_0^{\prime },S^{\prime },A^{\prime },{\left\{T_t^{\prime }\right\}}_{t=0}^{\infty }⟩\in \mathfrak{M}$, ${\rho }_0^{\prime }={\rho }_0$, $A^{\prime }=A$, which satisfies the following properties:

• $S^{\prime }=H$, where H is the history set of $\mathcal{N}$;
• $T_t^{\prime }:$$\begin{array}{ccc}\hfill S^{\prime }\times A^{\prime }& \to & {\Delta }_{S^{\prime }\times \mathbb{R}}\hfill \\ \hfill (h_t,a)& \mapsto & \left(T_t(h_t,a)\right)\circ (L_{S,t+1},1_{\mathbb{R}})\hfill \end{array}$.

Note that an MDP is a special case of an NMDP, which naturally induces the non-Markov embedding functor denoted as $\mathbf{N}\in \mathbf{Fun}(\mathfrak{M},\mathfrak{N})$. $\mathbf{M}$ and $\mathbf{N}$ are well-defined functors (Proof in Appendix C and Appendix D).

Definition 14 (Non-Markov Embedding).

Non-Markov embedding is a functor $\mathbf{N}:\mathfrak{M}\to \mathfrak{N}$. For any $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩\in \mathfrak{M}$, there exists ${\mathcal{N}}^{\prime }=\mathbf{N}\left(\mathcal{M}\right)=⟨{\rho }_0^{\prime },S^{\prime },A^{\prime },{\left\{T_t^{\prime }\right\}}_{t=0}^{\infty }⟩\in \mathfrak{N}$, ${\rho }_0^{\prime }={\rho }_0$, $S^{\prime }=S$, $A^{\prime }=A$, which satisfies the following property:

• $T_t^{\prime }:$$\begin{array}{ccc}\hfill H_t^{\prime }\times A^{\prime }& \to & {\Delta }_{S^{\prime }\times \mathbb{R}}\hfill \\ \hfill (h_t^{\prime },a)& \mapsto & (T_t\circ (L_{S,t},1_A))(h_t^{\prime },a)\hfill \end{array}$.

Although $\mathfrak{M}$ and $\mathfrak{N}$ have distinct properties by definition, we discover that they are equivalent categories. This leads to the following theorem (Proof in Appendix E).

Theorem 1 (Equivalence of $\mathfrak{M}$ and $\mathfrak{N}$).

Category $\mathfrak{M}$ and $\mathfrak{N}$ are equivalent through functor $\mathbf{M}$ and $\mathbf{N}$, i.e., $\mathbf{M}\circ \mathbf{N}\cong 1_{\mathfrak{M}}$, $\mathbf{N}\circ \mathbf{M}\cong 1_{\mathfrak{N}}$.

4. NMDP Constructed from MDP via HAS

Unlike many algorithmic studies that focus on developing functors in $\mathbf{Fun}(\mathfrak{N},\mathfrak{M})$ to handle NMDP’s history more effectively, our work focuses on constructing functors in $\mathbf{Fun}(\mathfrak{M},\mathfrak{N})$ for transforming MDPs into NMDPs. While $\mathbf{N}$ is such a functor, it is trivial since, in the constructed NMDP, the states prior to the current time are essentially irrelevant to decision-making. However, our interest lies in functors that can substantially incorporate non-Markovian transition dynamics into MDPs. To this end, we first introduce the History Aggregator for State (HAS), which enables us to construct such functors. Similarly, we can also define the History Aggregator for Reward (HAR), which substantially incorporates non-Markovian reward mechanisms into MDPs. As HAR is not the main content of this paper, we discuss it in the Appendix K. All proofs of theorems in this section are provided in Appendix F, Appendix G, Appendix H, Appendix I and Appendix J.

Definition 15 (History Aggregator for State (HAS)).

For any MDP $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$, an HAS $A_S:={\left\{A_{S,t}\right\}}_{t=0}^{\infty }$ is a series of maps in which $A_{S,t}:H_t\to A\left(H\right),(\forall t\in \mathbb{N})$ where $A\left(H\right)$ represents a set called the target state set of $A_S$.

As the name implies, the HAS constructs the NMDP state by aggregating the MDP history. Before detailing our method, we first require that the aggregators be reversible.

Definition 16 (Reversibility of HAS on MDP).

An HAS $A_S={\left\{A_{S,t}\right\}}_{t=0}^{\infty }$ on MDP $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ is reversible iff there exists a series of maps, denoted as $A_S^{\ast }:={\left\{A_{S,t}^{\ast }\right\}}_{t=0}^{\infty }$, which satisfy:

$$A_{S,t}^{\ast }\left({\left\{A_{S,\tau }\left(h_{\tau }\right)\right\}}_{\tau =0}^t\right)=s_t,(\forall t\in \mathbb{N})$$

where $h_{\tau }\in H_{\tau },h_0\prec h_1\prec \dots \prec h_t,s_t=L_{S,t}\left(h_t\right)$.

The reversibility condition of HAS ensures that a decision-making algorithm can reconstruct the original MDP state from the history of the NMDP. A reversible HAS-based transformation neither alters the transition dynamics of the original MDP nor increases its inherent complexity. Consequently, the performance upper bound for the NMDP is identical to that of the original MDP. Thus, once we know the best achievable performance of the original MDP, we can set the same expectations for the constructed NMDP. By ensuring that the transformation from the MDP to the NMDP does not increase the inherent difficulty of the original but only requires effective management of the NMDP’s history, we can more accurately evaluate a decision-making algorithm’s ability to handle non-Markovianity through encoding and memorization of NMDP’s history.

Noticing that the latest state extraction operators constitute reversible HAS, we have the following corollary.

Corollary 1.

The series of latest state extraction operators, $L_S:={\left\{L_{S,t}\right\}}_{t=0}^{\infty }$, constitutes a reversible HAS on MDP with a target state set identical to the state set.

When any reversible HAS is applied to an MDP, it generates an NMDP. The NMDP’s state set encompasses all aggregated histories of the original MDP. Its transition dynamics take the NMDP’s history as input, decode it into an MDP state, and then apply the MDP’s transition dynamics.

Definition 17 (Application of Reversible HAS to MDP).

The application of a reversible HAS $A_S={\left\{A_{S,t}\right\}}_{t=0}^{\infty }$ to an MDP $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ is an NMDP $\mathcal{N}=⟨{\rho }_0^{\prime },S^{\prime },A^{\prime },{\left\{T_t^{\prime }\right\}}_{t=0}^{\infty }⟩$, $A^{\prime }=A$ in which ${\rho }_0^{\prime }={\rho }_0\circ A_{S,0}^{\ast }$; $S^{\prime }=A\left(H\right)$, where H is the history set of $\mathcal{M}$, $H^{\prime }$ is the history set of $\mathcal{N}$;

$$T_t^{\prime }:\begin{array}{ccc}\hfill H_t^{\prime }\times A^{\prime }& \to & {\Delta }_{S^{\prime }\times \mathbb{R}}\hfill \\ \hfill (h_t^{\prime },a)& \mapsto & \left((T_t\circ (A_{S,t}^{\ast }\circ E_{S^{\prime }},1_A))(h_t^{\prime },a)\right)\circ \left(G_{S,h_t^{\prime }},1_{\mathbb{R}}\right)\hfill \end{array}$$

where $G_{S,h_t^{\prime }}:\begin{array}{ccc}\hfill S^{\prime }& \to & S\hfill \\ \hfill s_{t+1}^{\prime }& \mapsto & A_{S,t+1}^{\ast }\left(E_{S^{\prime }}\left(h_t^{\prime }\right),s_{t+1}^{\prime }\right)\hfill \end{array}$.

The decoding process, denoted as $A_{S,t}^{\ast }$ is the source of non-Markovianity. This is because, in the general case, decision-making algorithms have to remember the NMDP’s entire history to decode the MDP’s current state.

Although reversible HAS naturally induces functors in $\mathbf{Fun}(\mathfrak{M},\mathfrak{N})$, the definition of reversible HAS alone does not directly produce implementations, necessitating further specification.

For simplicity and practicality in implementation, we restrict our construction to using states alone, not the entire history. Given this constraint, we examine two types of HAS constructions: one using just states, and the other relying on auxiliary sequences.

4.1. HAS Constructed Using Just States

One straightforward way to construct HAS using just states is introducing a binary operator on the state set.

Definition 18 (HAS Induced by Binary Operator).

The HAS $B_S:={\left\{B_{S,t}\right\}}_{t=0}^{\infty }$ on MDP $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ induced by a binary operator $\otimes :S\times S\to S$ is:

$$B_{S,t}:\begin{array}{ccc}\hfill H_t& \to & S\hfill \\ \hfill h_t& \mapsto & \underset{\tau =0}{\overset{t}{⨂}}{s}_{\tau }\hfill \end{array}$$

where ${\left\{s_{\tau }\right\}}_{\tau =0}^t=E_S\left(h_t\right)$, ${⨂}_{\tau =0}^0{s}_{\tau }:=s_0$, and ${⨂}_{\tau =0}^{k+1}{s}_{\tau }:=\left({⨂}_{\tau =0}^k{s}_{\tau }\right)\otimes s_{k+1},(\forall k\in [0,t)\cap \mathbb{N})$.

According to the above definition, for the HAS induced by operator ⊗ to be reversible, it is sufficient for ⊗ to be a group operator, meaning $(S,\otimes )$ forms a group. This approach is quite general since any state set can be extended to form a group, like a free group. Moreover, in practical problems, states are typically represented as vectors in ${\mathbb{R}}^n$, and ${\mathbb{R}}^n$ forms a group under vector addition.

Theorem 2 (Reversibility of HAS Induced by Group Operator).

The HAS $G_S$ on MDP $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ induced by binary operator ⊗ is reversible if $(S,\otimes )$ is a group.

Let $\mathbf{G}\in \mathbf{Fun}(\mathfrak{M},\mathfrak{N})$ denote the functor induced by $G_S$. The proof of Theorem 2 in the Appendix F shows that $\mathbf{G}$ generates an NMDP. Unlike non-Markov embedding, the HAS $G_S$ does introduce non-Markovianity. It transforms an MDP, where decisions rely only on the current state, into an NMDP that requires knowledge of both the current and previous one-step state for decision-making. However, this state dependency structure is inadequate. Later, we’ll show that repeated application of this HAS can yield a more complex state dependency structure.

Definition 19 (State Dependency Structure of NMDP History).

The state dependency structure $D_{h_t}$ of a history $h_t=(s_{0:t},a_{0:t-1},r_{0:t-1})$ of NMDP $\mathcal{N}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ is a subset of $\mathbb{N}$

$$D_{h_t}:=\left\{i\mid \exists s\in S,\exists a\in A,T_t({\sigma }_i(h_t,s),a)\ne T_t(h_t,a)\right\}$$

where ${\sigma }_i(h_t,s)=((s_{0:i-1},s,s_{i+1:t}),a_{0:t-1},r_{0:t-1})$.

According to the above definition, $i\in D_{h_t}$ implies that $s_i$ has an impact on the state transition. Replacing it leads to a change in the subsequent transition probabilities. The concept of state dependency structure can also be extended to MDPs by defining it in terms of the state dependency structure of the MDP’s non-Markov embedding. It has been observed that applying $\mathbf{G}$ once to an MDP can increase the number of aggregated histories required to reconstruct the original MDP state in the resulting NMDP, thereby expanding the cardinality of the state dependency structure. Consequently, we explore applying $\mathbf{G}$ multiple times to the MDP to generate NMDPs with more complex temporal dependencies. To facilitate the application of reversible HAS to NMDPs, and without significant loss of generality, we assume that the transition dynamics of the NMDP depend solely on the state component of the history. The extension required to apply the reversible HAS to the NMDP is presented in the Appendix L. With a slight abuse of notation, we let $\mathbf{G}\in \mathbf{Fun}(\mathfrak{N},\mathfrak{N})$ also denote the functor induced by HAS.

Theorem 3 (Impact of $\mathbf{G}$ on State Dependency Structure).

For any MDP $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$, if it is non-degenerate, then the functor $\mathbf{G}$ induced by some group operator on S ensures that the dependency structure $D_{h_t^n}$ of any history $h_t^n\in H_t^n$ of ${\mathbf{G}}^n\left(\mathcal{M}\right)$ satisfies $D_{h_t^n}=[t-n,t]\cap \mathbb{N}$.

The significance of Theorem 3 lies in its provision of a general method for designing a transition mechanism for an NMDP that relies on the states of the previous $n+1$ steps. Specifically, it is sufficient to apply $\mathbf{G}$ functor n times to a non-degenerate MDP.

4.2. HAS Constructed Using Auxiliary Sequences

Another approach to aggregating states is using auxiliary sequences to assign importance weights to these states. In this way, a binary operator is required to connect an auxiliary sequence with the sequence of states.

Definition 20 (HAS Induced by Auxiliary Sequences and Binary Operator).

The HAS $W_S:={\left\{W_{S,t}\right\}}_{t=0}^{\infty }$ on MDP $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$ induced by a series of auxiliary sequences $W:={\left\{W_t\right\}}_{t=0}^{\infty }:={\left\{{\left\{w_{\tau }^t\right\}}_{\tau =0}^t\right\}}_{t=0}^{\infty }$ and a binary operator $\ast :W\times E_S\left(H\right)\to S$ is:

$$W_{S,t}:\begin{array}{ccc}\hfill H_t& \to & S\hfill \\ \hfill h_t& \mapsto & {\left\{w_{\tau }^t\right\}}_{\tau =0}^t\ast {\left\{s_{\tau }\right\}}_{\tau =0}^t\hfill \end{array}$$

where ${\left\{s_{\tau }\right\}}_{\tau =0}^t=E_S\left(h_t\right)$.

The binary operator ∗ does not specify how the auxiliary sequence elements interact with the state sequence elements. Therefore, we define two operators to determine this interaction, one operator $·:{⨆}_{t=0}^{\infty }{W}_t\times S\to S$ to apply an item in the auxiliary sequence to a state, the other operator $\oplus :S\times S\to S$ to aggregate two states. Without too much loss of generality, we restrict our discussion to left R-modules, where $(S,\oplus )$ forms an Abelian group, · is a scalar multiplication operator, and ${⨆}_{t=0}^{\infty }{W}_t$ becomes a unital ring with operators defined. We use $R_S$ to denote such an HAS.

Additionally, the correspondence between the elements of the auxiliary sequence and the elements of the state sequence needs to be specified. For the sake of simplicity in the subsequent discussion, we consider only two types of correspondence, where the auxiliary sequences are prefixes of a given sequence ${\left\{w_{\tau }\right\}}_{\tau =0}^{\infty }$: $\begin{array}{cccc}\hfill \overrightarrow{\ast }:& ({\left\{w_{\tau }\right\}}_{\tau =0}^t,{\left\{s_{\tau }\right\}}_{\tau =0}^t)& \mapsto & {⨁}_{\tau =0}^t(w_{\tau }·s_{\tau })\hfill \\ \hfill \overleftarrow{\ast }:& ({\left\{w_{\tau }\right\}}_{\tau =0}^t,{\left\{s_{\tau }\right\}}_{\tau =0}^t)& \mapsto & {⨁}_{\tau =0}^t(w_{\tau }·s_{t-\tau })\hfill \end{array}$. where $\overrightarrow{\ast }$ is referred to as correlation operator, and $\overleftarrow{\ast }$ is referred to as convolution operator.

We examine the reversibility of HAS in these two cases separately. For the correlation operator case, we have the following theorem:

Theorem 4 (Reversibility of HAS induced by Auxiliary Sequence and Correlation Operator).

The HAS $R_S:={\left\{R_{S,t}\right\}}_{t=0}^{\infty }$ induced by prefixes of auxiliary sequence ${\left\{w_t\right\}}_{t=0}^{\infty }$ and operator $\overrightarrow{\ast }$ is reversible if each $w_t$ is invertible in the ring.

Based on the proof in the Appendix H, it is obvious that the correlation operator only results in each set of the dependency structure of the constructed NMDP having a potential of no more than 2, which is similar to the effect of a group operator. Therefore, we focus on discussing the properties of the convolution operator.

For the convolution operator case, we have the following theorem:

Theorem 5 (Reversibility of HAS induced by Auxiliary Sequence and Convolution Operator).

The HAS $R_S:={\left\{R_{S,t}\right\}}_{t=0}^{\infty }$ induced by prefixes of auxiliary sequence ${\left\{w_t\right\}}_{t=0}^{\infty }$ and operator $\overleftarrow{\ast }$ is reversible if $w_0$ is invertible in the ring.

We use $\mathbf{R}\in \mathbf{Fun}(\mathfrak{M},\mathfrak{N})$ to denote the functor induced by reversible $R_S$ and operator $\overleftarrow{\ast }$, then we have the following theorem about its impact on the state dependency structure of the NMDP.

Theorem 6 (Impact of $\mathbf{R}$ on State Dependency Structure).

For any MDP $\mathcal{M}=⟨{\rho }_0,S,A,{\left\{T_t\right\}}_{t=0}^{\infty }⟩$, if it is non-degenerate, then the functor $\mathbf{R}$ induced by prefixes of auxiliary sequence ${\left\{w_{\tau }\right\}}_{\tau =0}^{\infty }$ and operator $\overleftarrow{\ast }$ ensures that the dependency structure $D_{h_t^{\prime }}$ of any history $h_t^{\prime }\in H_t^{\prime }$ of $\mathbf{R}\left(\mathcal{M}\right)$ satisfies $D_{h_t^{\prime }}=\{t-\tau \mid {\left({\mathbf{w}}^{-1}\right)}_{0,\tau }\ne 0\}$, where ${\mathbf{w}}^{-1}$ is the inverse matrix of $\mathbf{w}$ in the ring, 0 is the zero element of the ring.

The proof of Theorem 6 indicates that it is possible to achieve any specific state dependency structure in the history of the constructed NMDP by modifying the elements of the upper triangular band matrix ${\mathbf{w}}^{-1}$. This provides a general method for designing NMDPs with desired state dependency structures.

5. Experiments

Although the main contribution of this paper is theoretical, we have designed and implemented several highly versatile environment wrappers based on the two methods proposed in Section 4 for constructing functors in $\mathbf{Fun}(\mathfrak{M},\mathfrak{N})$: the group-operator-based and the convolution-based method. These wrappers are implemented using a computationally efficient approach with a time complexity of $O\left(n\right)$ and can be applied to environments that conform to the Gymnasium [20] interface.

We use the PPO [21] algorithm implemented in the Stable-Baselines3 library [22] and the LSTM-PPO algorithm of the Stable-Baselines3-Contrib library [22] as standard reinforcement learning algorithms. These algorithms are tested in Gymnasium environments wrapped with our wrappers, with reinforcement learning training experiments conducted using the framework provided by the RL-Baselines3-Zoo library [23].

We assume $s_{-1}:=\mathbf{0}$, $\lambda \in [0,1]$, $\exists k\forall t,s_t\in {\mathbb{R}}^k$, and employ the following functor to implement wrappers:

• $\mathbf{S}$ induced by HAS $S_{S,t}:h_t\mapsto {\sum }_{\tau =0}^t{s}_{\tau }$;
• $\mathbf{D}$ induced by HAS $D_{S,t}:h_t\mapsto s_t-s_{t-1}$;
• ${\mathbf{S}}_{\lambda }$ induced by HAS $S_{S,t}^{\lambda }:h_t\mapsto {\sum }_{\tau =0}^t{\lambda }^{\tau }{s}_{t-\tau }$;
• ${\mathbf{D}}_{\lambda }$ induced by HAS $D_{S,t}^{\lambda }:h_t\mapsto s_t-\lambda s_{t-1}$.

From the definitions of $\mathbf{S}$, $\mathbf{D}$, ${\mathbf{S}}_{\lambda }$, ${\mathbf{D}}_{\lambda }$, we can derive the state dependency structure of histories in the NMDP under their application. $\mathbf{S}$, ${\mathbf{S}}_{\lambda }$ introduce dependencies of $s_t$, $s_{t-1}$, and $\mathbf{D}$, ${\mathbf{D}}_{\lambda }$ introduce dependencies of $s_{0:t}$. The state dependency weights induced by ${\mathbf{S}}_{\lambda }$ and ${\mathbf{D}}_{\lambda }$ exhibit an exponential decay with respect to the temporal distance from the current time step, governed by the factor $\lambda$, while $\mathbf{S}$ and $\mathbf{D}$ introduce uniformly distributed dependency.

Our experiments use CartPole-v1 and Pendulum-v1 as the base MDP environments, and we employ the following functors to obtain the NMDP environments:

$${\mathbf{S}}^n,{\mathbf{D}}^n \mathrm{for} n\in \{0,1,2,3,4,5\}$$

$${\mathbf{S}}_{\lambda },{\mathbf{D}}_{\lambda } \mathrm{for} \lambda \in \{{\displaystyle \frac{0}{5}},{\displaystyle \frac{1}{5}},{\displaystyle \frac{2}{5}},{\displaystyle \frac{3}{5}},{\displaystyle \frac{4}{5}},{\displaystyle \frac{5}{5}}\}$$

All hyperparameters required for training were kept at their default settings from the RL-Baselines3-Zoo library, and the training process was initiated using the library’s provided command-line instructions. Each combination of environment, wrapper, and algorithm was trained 3 times, ${10}^6$ timesteps for each time, evaluated every $2\times {10}^5$ timesteps, and the checkpoints with the best average episode reward were selected.

Theoretically, an increase in n of ${\mathbf{S}}^n$ and ${\mathbf{D}}^n$ or $\lambda$ of ${\mathbf{S}}_{\lambda }$ and ${\mathbf{D}}_{\lambda }$ makes the non-Markov environment more challenging for RL algorithms to solve, as it introduces more complex temporal dependencies. As shown in Figure 1, the results of our experiments also indicate this property: the average episode reward, plotted as a function of n or $\lambda$, generally shows a declining trend for each combination of environment, algorithm, and wrapper.

Figure 1. Experimental results for each combination of environment, algorithm, and wrapper. (a) CartPole-v1 with ${\mathbf{S}}^n,{\mathbf{D}}^n$ wrappers; (b) CartPole-v1 with ${\mathbf{S}}_{\lambda },{\mathbf{D}}_{\lambda }$ wrappers; (c) Pendulum-v1 with ${\mathbf{S}}^n,{\mathbf{D}}^n$ wrappers; (d) Pendulum-v1 with ${\mathbf{S}}_{\lambda },{\mathbf{D}}_{\lambda }$ wrappers; The x-axis represents the wrapper parameter, while the y-axis shows the average episode reward. Lines of different colors indicate different combinations of algorithms and wrapper types.

Another conclusion drawn from the experimental results is that, for any given combination of environment and algorithm, the performance degradation caused by increasing $\lambda$ from 0 to 1 is comparable to the degradation resulting from increasing n from 0 to 1. This observation is consistent with our theoretical analysis, which shows that increasing $\lambda$ from 0 to 1 progressively introduces a greater dependence on previous states, eventually matching the dependence introduced by a wrapper with $n=1$.

By comparing the performance of the PPO and LSTM-PPO algorithms in environments wrapped with varying degrees of dependency complexity, we observe that LSTM-PPO outperforms PPO when the dependencies are more complex. This is due to LSTM-PPO’s ability to recall past states, which also aligns with our intuition.

5.1. Comparative Analysis of the Proposed Framework

To systematically illustrate the advantages of our proposed framework and how it addresses the limitations of existing studies, we provide a comparative analysis in Table 2. Our approach, grounded in category theory and reversible HAS, offers a unified, flexible, and theoretically sound methodology for constructing non-Markovian decision processes, overcoming the ad-hoc, information-lossy, and unsystematic nature of prior methods.

Table 2. Comparison between existing studies and the proposed framework on constructing non-Markovian decision processes.

Weaknesses/Limitations in Existing Studies	Advantages/Improvements in Our Proposed Framework
Lack of a unified theoretical foundation. Existing models (e.g., RDP, RM, Markov Abstraction) vary in formulation and lack a common framework for comparison.	Category-theoretic unification. We establish categories $\mathfrak{M}$ and $\mathfrak{N}$, proving their equivalence, thereby providing a unified foundation for MDP/NMDP transformations.
Ad-hoc and domain-specific environment construction. Non-Markovian benchmarks are often built via simple masking or domain-specific simulators, leading to scattered and incomparable evaluations.	Systematic and general construction via HAS. Our HAS-based method allows controlled introduction of non-Markovianity into any MDP, enabling systematic generation of benchmarks with precise dependency structures.
Information loss in POMDP-based conversions. Many approaches create non-Markovianity by masking observations, which alters the inherent problem difficulty and may confound evaluation.	Reversibility preserves problem essence. Reversible HAS ensures the original MDP state can be recovered from NMDP history without information loss, allowing pure evaluation of memory/decoding abilities.
Limited control over temporal dependency structures. Existing methods do not offer fine-grained control over how many past states influence the current transition.	Tunable state dependency structures. Through repeated application of $\mathbf{G}$ or design of auxiliary sequences in $\mathbf{R}$, we can precisely control the length and weights of historical dependencies.
No standardized interface for algorithm testing. Environments built by different studies are incompatible, hindering reproducible and comparable algorithm evaluation.	Compatible environment wrappers. Our implementations provide Gymnasium-compatible wrappers, allowing seamless testing of the same algorithm on both MDP and constructed NMDP versions.
Difficulty in isolating non-Markovian challenges. Many benchmarks conflate non-Markovianity with other complexities (e.g., partial observability, stochastic dynamics).	Pure non-Markovian challenge via reversible HAS. By preserving transition dynamics and only altering state representations, we isolate the challenge of history memorization and decoding.

As demonstrated in Table 2, our framework not only overcomes key weaknesses in prior art but also provides a flexible, theoretically grounded, and practical toolkit for constructing non-Markovian decision problems, thereby facilitating more rigorous and comparable evaluation of decision-making algorithms.

5.2. Practical Implications and Target Audience

The proposed framework and experimental results are designed to serve a broad spectrum of researchers and practitioners working on decision-making under temporal dependencies. Specifically, the following groups are poised to benefit most from our work:

• Reinforcement Learning Researchers studying non-Markovian environments, memory-based architectures, or partial observability can use our HAS-based construction method to generate benchmark environments with controllable and reversible temporal dependencies. This allows for systematic ablation studies on how different dependency structures affect algorithm performance, without confounding factors such as information loss or altered task difficulty.
• Algorithm Developers aiming to evaluate the robustness of RL algorithms (e.g., PPO, LSTM-based agents) in non-Markovian settings can adopt our Gymnasium-compatible wrappers to seamlessly convert existing Markovian benchmarks into non-Markovian variants. Our results provide a clear reference: algorithms with memory mechanisms (like LSTM-PPO) consistently outperform memory-less counterparts as dependency complexity increases, underscoring the importance of architectural choices in such settings.
• Decision System Designers in Applied Fields (e.g., robotics, healthcare, finance) where real-world dynamics often exhibit non-Markovian characteristics can leverage our framework to “stress-test” their policies in progressively more realistic environments. By tuning parameters such as n in ${\mathbf{S}}^n$ or $\lambda$ in ${\mathbf{S}}_{\lambda }$, they can simulate varying degrees of historical influence, thus identifying the memory capacity required for robust decision-making in their specific domain.

In summary, our work provides not only a theoretical foundation but also a practical toolkit for constructing, analyzing, and benchmarking decision processes beyond the Markovian assumption, thereby supporting more rigorous and generalizable progress in algorithmic decision-making.

6. Summary and Future Work

We propose a general and effective method to construct NMDPs from MDPs using techniques from category theory and algebra. Our method provides a comprehensive way to evaluate decision-making, especially RL algorithms’ ability to handle non-Markovianity through encoding and memorization.

Theoretical analysis demonstrates that our method possesses strong expressive power and can represent a wide variety of temporal state dependency structures. Using this theoretical framework, we transformed classical Markovian environments into non-Markovian ones and tested them with both PPO algorithms, with and without LSTM. This validation confirms our method’s effectiveness in introducing varying degrees of non-Markovian characteristics into Markovian environments.

The examples tested in our experiments represent only a small subset of the cases within the theoretical framework. The functor forms that exhibit desirable properties through our theory are highly diverse, extending beyond those tested. This is due to our proofs being conducted within abstract algebraic structures, which encompass numerous notable specific cases.

Our study also has limitations that warrant further investigation. For instance, we did not explore the introduction of randomness into HAS using random sequences or mappings. Additionally, we did not address the theory of HAR, which remains relevant for reward shaping. Moreover, our theoretical analysis is restricted to reversible HAS, and we have not developed a theory for non-reversible cases. For example, a common method to create non-Markovian environments is by obscuring information in the observations of an MDP to transform it into a POMDP. Combining this method with our approach and exploring their combined expressive power presents a promising avenue for future research.

In future work, we also plan to leverage the HAS framework and Gymnasium-compatible wrappers presented here to establish an open-source benchmark suite for non-Markovian decision processes. This suite will facilitate systematic, reproducible comparisons of various methods, including Reward Machines, Regular Decision Processes, and POMDP conversions, against our HAS-based construction, empirically validating its advantages in generating environments with controllable dependencies and no inherent information loss.