Undiscounted Semi-Markov Decision Processes with Countably Infinite Action Spaces

Abstract

In this article, we study semi-Markov decision processes (SMDPs) under the limiting ratio average (undiscounted) pay-off criterion, where the state space is finite and the action space of the decision maker is possibly countably infinite. We impose no restriction on the reward function. We prove that the value of such an SMDP exists and that the decision maker possesses a near-optimal ($ϵN$-optimal) deterministic (pure) semi-stationary strategy, and we give a truncation-based algorithm that computes the value and such a strategy in finite time by solving a finite linear program at each truncation level. We analyze the algorithm’s computational complexity, showing that each truncation level is solved in time polynomial in the number of state–action pairs, while the number of levels required for termination depends on the prescribed accuracy and on the model. We further establish a certified, computable bound on the truncation error under bounded rewards together with standard ergodicity and tail-regularity conditions, yielding a stopping rule that guarantees a prescribed accuracy; we delimit precisely the regime in which such a certificate exists. Numerical experiments, including randomly generated instances, illustrate the algorithm and its computational behavior. Finally, under standard ergodicity conditions, we derive an optimality equation for these SMDP models.

1. Introduction

The study of Semi-Markov Decision Processes (SMDPs) was initiated by Jewell [1] and Howard [2], and was subsequently extended and generalized by several researchers, including Ross [3], Lippman [4], and Federgruen and Tijms [5]. Over time, the framework has evolved significantly, both in terms of theoretical development and practical applicability. For instance, Xiaobo et al. [6] investigated an SMDP model under an alternative payoff criterion, leading to a broader and more flexible class of admissible strategies for the decision maker.

SMDPs provide a natural extension of classical Markov Decision Processes by allowing the sojourn time—the time spent in a state before transitioning—to be a random variable that depends not only on the current state and chosen action, but also on the subsequent state. This added layer of realism makes SMDPs particularly suitable for modelling stochastic control problems where timing plays a critical role.

In recent years, research in this area has increasingly focused on integrating SMDPs with modern computational paradigms and applying them to complex, real-world decision-making problems. In particular, under the limiting ratio average payoff criterion, the intersection of SMDPs with reinforcement learning (RL) has gained considerable attention. For example, Fatemi et al. [7] proposed a framework that incorporates SMDP structures into value-based offline RL methods, with applications in healthcare systems. Similarly, Boucher et al. [8] developed an SMDP-based approach within learning-based model predictive control for dynamical systems. Stephenson et al. [9] demonstrated how semi-Markov decision models can be used to enhance the operational efficiency of agile Earth-observing satellites.

In parallel, reinforcement-learning-based optimal-control methods for nonlinear and multi-agent systems have advanced rapidly, including SMS-based event-triggered fault-tolerant control of nonlinear systems via adaptive dynamic programming [10]; and filter-based predefined-time optimal fault-tolerant consensus control for nonlinear multi-agent systems via reinforcement learning [11]. These developments share with our work the use of average-reward/optimal-value criteria, and motivate the study of SMDP foundations on which such learning methods rest.

Beyond these developments, SMDPs have found wide-ranging applications across several domains. In aerospace engineering, for instance, they have been employed to predict the remaining useful life (RUL) of aircraft components, as studied by Diyin et al. [12]. Additional applications and methodological advancements can be found in [13,14,15,16].

In the context of healthcare systems, SMDPs are particularly effective in modelling patient progression through different stages of treatment, where transitions often follow semi-Markov dynamics. Such models help in designing policies that not only improve patient outcomes but also optimise healthcare costs. Recent contributions in this direction include [17,18,19].

Overall, the growing body of literature highlights the versatility of SMDPs and their increasing relevance in addressing both theoretical and applied problems in stochastic decision-making.

In this paper, we investigate undiscounted semi-Markov decision processes (SMDPs) with countably infinite action spaces where the state space is finite and the action space of the decision maker is (possibly) countably infinite. Such models arise in real-life decision making problems, where the set of feasible actions is not finite or unbounded in nature. Unlike classical SMDP model, where we assume that the state space and admissible action space are typically finite, we allow the action space to be countably infinite, which introduces additional computational complexity to establish the existence of optimal strategies. Here, we assume that the decision maker has access to a countably infinite set of admissible actions in each state. However, if there is any state where the number of actions is finite, then we can repeat a particular cell in that state to make the action space of that state countably infinite. We made no restrictions on the rewards. By undiscounted pay-offs, here, we mean limiting ratio average pay-offs.

From a modelling standpoint, the semi-infinite SMDPs are useful when the decision maker has a countably infinite or an indeterminate set of feasible actions. This can be illustrated by the following cases:

(i)

In inventory control with unbounded replenishment, a manager may place a replenishment order of any non-negative quantity, which makes the admissible action set in each state countably infinite. The sojourn time is state-action dependent, as the lead time before arriving stock is random. Thus, we can interpret the problem as an SMDP model.

(ii)

In the control of queueing systems, the controller may choose a service rate among a countably infinite family of service configurations (e.g., the number of activated servers $k=1,2,3,\dots$), where a higher index gives faster service at greater cost. Here, the service completion time is a non-exponential random variable. Thus, we can also think of this problem in an SMDP framework.

In both cases, the cardinality of the action set is not a priori bounded, which provides the motivation to study the countably semi-infinite SMDP framework.

We prove that in such a model, the value of the decision process exists and the decision maker has near-optimal pure semi-stationary strategies. We have shown this result by exploiting the results of Sinha et al. [20], as the decision process can be regarded as a one-player game. Thus, the results can be applied for an undiscounted SMDP having finite states and countably many actions. We will refer to such SMDP as an infinite SMDP model. From the point of view of modelling real phenomena, the model with countably many actions can be used in a situation where the decision-maker has finitely many actions, but the cardinality of that set is not known appropriately. From a modeling standpoint, infinite SMDPs are particularly useful in situations where the decision maker effectively operates over a finite but indeterminate or evolving set of actions. For example, in adaptive control, inventory systems with flexible policies, or queueing systems with dynamic service configurations, the number of available actions may not be explicitly bounded or may depend on external conditions. Thus, the countable action framework provides both generality and modeling flexibility. In this paper, we consider a maximization problem for the infinite SMDP. The paper is organized as follows. Section 2 contains the preliminaries and notations of an undiscounted infinite SMDP. In Section 3, we define the undiscounted (limiting ratio average) pay-off criterion of an SMDP. In Section 4, we define the $ϵN$-(near-)optimal strategy of the decision maker in an infinite SMDP and prove that the value of such an SMDP model exists, together with the existence of near-optimal pure semi-stationary strategies. Section 5 develops the algorithmic and computational contributions: we propose a truncation-based algorithm to compute the value and a near-optimal pure semi-stationary strategy (Section 5.1), analyze its complexity and convergence (Section 5.2), and establish a certified computable error bound on the truncation error together with the corresponding stopping rule (Section 5.5 and Section 5.6); numerical examples, including a worked example and randomly generated instances, are presented within this section. The optimality equation of such an SMDP model, derived under recurrence/ergodicity conditions, is given in Appendix A. Section 6 is reserved for the conclusion.

2. Preliminaries and Notations

  An undiscounted semi-Markov decision process with countably infinite action space, considered in this process, is denoted by ${\Gamma }_{\infty }=<S,\{A\left(s\right):s\in S\}$, $\left\{r\right(s,a):s\in S,$$a\in A\left(s\right)\},\{q(s\prime \mid s,a):s\in S,a\in A\left(s\right)\},\{\overline{\tau }(s,a):s\in S,a\in A\left(s\right)\}>$ which consists of a finite state space $S=\{1,2,\cdots ,z\}$. For each $s\in S$, we associate a countable set of actions $A\left(s\right)=\{1,2,\dots \}$. The process is controlled by a decision maker. When the system is in the state s, the controller chooses an action $a\in A\left(s\right)$ and gets an immediate reward $r(s,a)$. Let us denote $K=\left\{\right(s,a):s\in S,a\in A(s),\}$ to be the set of admissible tuples. For each $(s,a)\in K$, we denote $q(.\mid s,a)$ to be the transition law of the game. Given $(s,a)\in K$ and $s\prime \in S$, let ${\tau }_{ss\prime }^a$ be the transition time random variable which denotes the time for a transition to a state $s\prime$ from a state s by the action $a\in A\left(s\right)$. Let $P_{ss\prime }^a=Prob({\tau }_{ss\prime }^a\le t)$ for each $(s,a)\in K,s\prime \in S$ be a probability distribution function on [$0,\infty$); it is called the conditional transition time distribution function. The process moves to a state $s\prime \in S$ with probability $q(s\prime \mid s,a)$, where $q(s\prime \mid s,a)\ge 0$ and ${\sum }_{s\prime \in S}q(s\prime \mid s,a)=1$.

The SMDP model evolves over infinite time horizon as below:

At the initial decision epoch, the process begins at a state $s_0\in S$. The decision maker observes $s_0$ and selects an action $a_0\in A\left(s_0\right).$ As an effect of this, the decision maker gets an immediate reward $r(s_0,a_0)$. At the next decision epoch, the system transits to a new state $s_1\in S$ with probability $q(s_1\mid s_0,a_0)$. Let, $P{r}_{s_0{s}_1}^{a_0}(.)$ denote the transition time distribution function that determines the sojourn time taken from the state $s_0$ to state $s_1$, when the decision maker chooses action $a_0$ at state $s_0$. After the transition to the state $s_1$, the same mechanism repeats for subsequent states and actions. The process thus continues infinitely over time. The decision maker in this model has ‘unbounded memory’ in the sense that he/she can recall at any time the states, actions and transition times of the past, which make an SMDP model, a process with perfect recall (Kuhn [21]).

It is evident that when transition times are identical, an SMDP model is reduced to an MDP (Markov Decision Process).

Here, the strategies of the decision maker are restricted to strategies with finite support, i.e., for each $\pi \in \Pi$ there is k$\in \mathbb{N}$, such that with probability 1 the decision maker chooses one of the first k cells in all states, for all histories. Mathematically, the decision maker chooses mixed actions from ${\cup }_{k\in \mathbb{N}}{A}_k$, where $A_k=\{a\in {\mathbb{R}}^{\infty }:a\ge 0,{\sum }_{i=1}^k{a}_i=1;a_i=0$ for $i>k\}$.

We now discuss the definitions of strategy spaces of the decision maker in an undiscounted SMDP, where the state and the action space are finite. A strategy is labeled as a behavioral strategy of the decision maker, is denoted by $\pi$. By this, we mean a sequence ${\left\{\left({\pi }_n\right)(.\mid his{t}^n)\right\}}_{n=0}^{\infty }$, where $\left({\pi }_n\right)$ determines the action chosen by the decision maker at n-th day. This is done by mapping it with each history $his{t}^n$ of the system upto nth day (The history upto n-th day is denoted by $his{t}^n$=$(s_0,a_0,,s_1,a_1,\cdots ,s_{n-1},a_{n-1},s_n)$ for $n\ge 1$, $his{t}^0=\left(s_0\right)$ and $(s_k,a_k)\in {\kappa }_a$ are respectively the state and actions of the decision maker at the k-th decision epoch) to a probability distribution ${\left(\pi \right)}_n(.\mid his{t}^n)$ on $A\left(s_n\right)$. Let us denote $\Pi$ to be the set of behavioral strategies of the decision maker.

Semi-Markov strategy is denoted by $m={\left\{m_n\right\}}_{n=0}^{\infty }$ and for each n, $m_n$ depends on the initial state $s_0$, the current state $s_n$ and the decision epoch number n.

Stationary strategy is generally denoted by $f_s={\left\{f_n\right\}}_{n=0}^{\infty }$, which is a map $f_s:S\to \mathbb{P}\left(A\right)$, such that (i) $f_n=f_s$$\forall n\in \{0,\cdots ,\infty \}$, i.e., $f_n(.\mid s_0,s_1,\cdots ,s_n=s)=f_s\left(s\right)$ also (ii) $f_s\left(s\right)\in \mathbb{P}\left(A\left(s\right)\right)$ for each $s\in S$. We denote $F^s$ as the set of stationary strategies of the decision maker.

A pure stationary strategy is a stationary strategy in which a player selects a particular action while visiting a state s with probability 1 or probability 0. Let $F^{sp}$ denotes the pure stationary strategy set of the decision maker.

A strategy $g_{sp}={\left\{{g_{sp}}_n\right\}}_{n=1}^{\infty }$ is called semi-stationary strategy if there exists a map ${\zeta }_{sp}:S\times S\to \mathbb{P}\left(A\right)$ such that (i) ${g_{sp}}_n(.\mid s_0=s,\cdots ,s_n=s\prime )={\zeta }_{sp}(s,s\prime )$ for all $s_0,s_1,\cdots ,s_n\in S$, $n\in \{0,\cdots ,\infty \}$ and (ii) ${\zeta }_{sp}(s,s\prime )$ has a support in $A(s\prime )$ for each pair $(s,s\prime )\in S\times S$. So, we can think of a semi-stationary strategy as a semi-Markov strategy which is independent of the day number n, i.e., for an beginning state $s_0$ and current state $s_1$, if a semi-Markov strategy $g_{sp}(s_0,s_1,n)$ has no dependency on n, then we label it as a semi-stationary strategy.

Let ${\xi }^s$ and ${\xi }^{sp}$ denote the set of semi-stationary and pure semi-stationary strategies respectively for the decision maker.

For each undiscounted SMDP with countably infinite action space, defined as ${\Gamma }_{\infty }$, we associate a truncated SMDP ${\Gamma }_n$, where ${\Gamma }_n$ is defined as ${\Gamma }_n=<S,\{A_n\left(s\right):s\in S\}$, $\{r(s,a):s\in S,a\in A_n\left(s\right)\},\{q(s\prime \mid s,a):s\in S,a\in A_n\left(s\right)\}>$ where $A_n\left(s\right)\subset A\left(s\right)$ consists of the actions for which $a<n$. Thus, the truncated game can be found from $\Gamma$ by deleting all cells in $\Gamma$ for which $a>n$ where $n\in \mathbb{N}$. We show that the value of the infinite SMDP model exists in ${\mathbb{R}}_{+}^z,$ where ${\mathbb{R}}_{+}^z={(\mathbb{R}\cup \{+\infty \})}^z$. In an undiscounted infinite SMDP, any strategy with finite support $n\in \mathbb{N}$ can be identified with the strategies in ${\Gamma }_n$. By completion with zeros in the cells of the n-truncated SMDP, the strategies for the decision maker in n-truncated SMDP ${\Gamma }_n$, can be identified with finite support strategies of the decision maker in ${\Gamma }_{\infty }$. Similarly, by deleting zeros from the cells of the infinite SMDP, finite support strategies of the decision maker in ${\Gamma }_{\infty }$ can be identified with strategies of the decision maker in ${\Gamma }_n$ for sufficiently large n.

3. Undiscounted Pay-Off Criterion in an Infinite SMDP Model

Suppose $\Gamma =<S,A,q,P,r>$ is an undiscounted SMDP with finite state space denoted as S and finite action space denoted as A. Further, we denote $(X_1,A_1,X_2,A_2,\cdots )$ as a sequence of coordinates in $S\times {(A\times S)}^{\infty }$. Given a (behavioral) strategy $\pi \in \Pi$ and an initial state $s\in S$, there is a probability measure (which is unique) ${\mathbb{P}}_{\pi }(.\mid X_1=s)$ (and therefore an expectation ${\mathbb{E}}_{\pi }(.\mid X_1=s)$) on the product $\sigma$-field of $S\times {(A\times S)}^{\infty }$. We have this corollary from Kolmogorov’s extension theorem. The limiting ratio average pay-off for the decision maker for a behavioral strategy $\pi \in \Pi$ is defined as:

$${\varphi }_1(s,\pi )=\underset{n\to \infty }{lim\; inf}{\displaystyle \frac{{\mathbb{E}}_{\pi }{\sum }_{m=1}^n[r(X_m,A_m)\mid X_1=s]}{{\mathbb{E}}_{\pi }{\sum }_{m=1}^n[\overline{\tau }(X_m,A_m)\mid X_1=s]}}\mathrm{for}\mathrm{all}s\in S.$$

(1)

and

$${\varphi }_2(s,\pi )=\underset{n\to \infty }{lim\; sup}{\displaystyle \frac{{\mathbb{E}}_{\pi }{\sum }_{g=1}^n[r(X_g,A_g)\mid X_1=s]}{{\mathbb{E}}_{\pi }{\sum }_{g=1}^n[\overline{\tau }(X_g,A_g)\mid X_1=s]}}\mathrm{for}\mathrm{all}s\in S.$$

(2)

The expected sojourn time here is defined by:

$$\overline{\tau }(s,a)=\sum _{s\prime \in S}q(s\prime \mid s,a){\int }_0^{\infty }tdP{r}_{ss\prime }^a\left(t\right).$$

(3)

We assume that $\overline{\tau }(s,a)$ is bounded away from zero for each $(s,a)\in K$, where $K=\left\{\right(s,a):s\in S,a\in A(s\left)\right\}$.

Definition 1.

A ${\pi }^{*}\in \Pi$ under limiting ratio average pay-off is called optimal if:

$${\varphi }_1(s,{\pi }^{*})\ge {\varphi }_1(s,\pi )(({\varphi }_2(s,{\pi }^{*})\ge {\varphi }_2(s,\pi ))\mathrm{for}\mathrm{all}\pi \in \Pi \mathrm{and}\mathrm{all}s\in S.$$

(4)

Let $v_1\left(s\right)={sup}_{s\in S}\left({\varphi }_1(s,\pi )\right)$ and $v_2\left(s\right)={sup}_{s\in S}\left({\varphi }_2(s,\pi )\right)$. Then, $v_1\left(s\right)$ (resp. $v_2\left(s\right))$ is called the limiting ratio average value of the undiscounted SMDP for the initial state $s\in S$. Let $f\in F^s$ be a stationary strategy and $s\in S$ be any arbitraty state. Then, $r(s,f)={\sum }_{a\in A\left(s\right)}r(s,a).f(s,a)$, $\overline{\tau }(s,f)={\sum }_{a\in A\left(s\right)}\overline{\tau }(s,a)f(s,a)$ and $q(s\prime \mid s,f)={\sum }_{a\in A\left(s\right)}q(s\prime \mid s,a)f(s,a)$. Consequently, the reward vector, sojourn time vector and transition probability matrix can be defined respectively as: $r\left(f\right)={\left[r(s,f)\right]}_{z\times 1}$, $\overline{\tau }\left(f\right)={\left[\overline{\tau }(s,f)\right]}_{z\times 1}$ and $Q\left(f\right)={\left[q(s\prime \mid s,f)\right]}_{z\times z}$. Let $Q^m\left(f\right)={\left[q^m(s\prime \mid s,f)\right]}_{z\times z}$ where $q^m(s\prime \mid s,f)$ is the m-step transition probability from state s to $s\prime$ under the stationary strategy f.

The following lemma from Doob [22] illustrates how to get $q^{*}(s\prime \mid s)$ from the m-step transition probability $q^m(s\prime \mid s)$.

Lemma 1.

Let $Q={\left[q(s\prime \mid s)\right]}_{z\times z}$ be a matrix of transition probabilities. Then ∃ a matrix $Q^{*}={\left[q^{*}(s\prime \mid s)\right]}_{z\times z}$ (whose row sums are 1), which is called the Cesaro-limiting matrix of Q, is defined as:

$$\underset{n\to \infty }{lim}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{m=1}^n}{q}^m(s\prime \mid s)=q^{*}(s\prime \mid s)s,s\prime \in S$$

(5)

Following the above lemma and Sinha et al. [23], we have the following important result:

Theorem 1.

Let $f\in F^s$ be a stationary strategy, then:

$$\varphi (s,f)={\displaystyle \frac{\left[Q^{*}\left(f\right)r\left(f\right)\right]\left(s\right)}{\left[Q^{*}\left(f\right)\overline{\tau }\left(f\right)\right]\left(s\right)}}\mathrm{for}\mathrm{all}s\in S$$

(6)

where $Q^{*}\left(f\right)$ is the Cesaro-limiting matrix of the transition matrix $Q\left(f\right)$ and $\varphi (s,f)$ is the undiscounted pay-off of the decision maker.

For a stationary strategy $f\in F^s$ and all $s\in S$, we denote the limiting ratio average pay-off and the value of the n-truncated SMDP associated with an infinite undiscounted SMDP as ${\varphi }_n(s,f)$ and $v_n$. Also, for a stationary strategy $f\in F^s$ (with finite support) and for all $s\in S$, the pay-off function (limiting ratio average pay-off) for the undiscounted SMDP with countable actions is denoted as ${\varphi }_{\infty }(s,f)$.

4. $\mathit{ϵ}\mathit{N}$-Optimal Strategies for the Decision Maker in an Infinite Undiscounted SMDP Model

  In this section, we define the $ϵN$-optimal or near-optimal strategies for the decision maker in an infinite undiscounted SMDP model, assuming the infinite SMDP is a maximization problem.

Definition 2.

A strategy ${\pi }_{ϵN}^{*}$ is called an $ϵN$-optimal strategy if there exists a $v_{\infty }\in {\mathbb{R}}_{+}^z$ such that for all $ϵ>0$ and $N>0(\in \mathbb{N})$ and for all $s\in S$, the following inequality holds:

$${\varphi }_{\infty }(s,{\pi }_{ϵN}^{*})\ge \left\{\begin{array}{cc}{v}_{\infty }\left(s\right)-ϵ\hfill & \hfill \mathrm{if}{v}_{\infty }\left(s\right)\in \mathbb{R}\\ N\hfill & \hfill \mathrm{if}{v}_{\infty }\left(s\right)=+\infty \end{array}\right.$$

(7)

Then, $v_{\infty }$ is called the value vector of the SMDP ${\Gamma }_{\infty }$.

Now, we prove the following Theorem to establish the fact that there exists an $ϵN$-optimal pure semi-stationary strategy of the decision maker in the undiscounted SMDP model.

Theorem 2.

(i) 

${lim}_{n\to \infty }{v}_n$ exists and equals to $v_{\infty }$.

(ii) 

The decision maker has a near-optimal pure semi-stationary strategy in ${\Gamma }_{\infty }$.

Proof. 

We know from Sinha et al. [23], for an undiscounted (limiting ratio average) SMDP, the value and optimal pure semi-stationary strategy exists. Now the semi-stationary strategy space of ${\Gamma }_{n+1}$ contains that of ${\Gamma }_n$. Thus the value vector $v_{n+1}$, i.e., the maximal value in ${\Gamma }_{n+1}$, will lie above $v_n$, that of ${\Gamma }_n$. So, ${\left\{v_n\right\}}_{n=1}^{\infty }$ is an increasing sequence and hence it converges to a point $w\in {\mathbb{R}}_{+}^z$ coordinate-wise (where ${\mathbb{R}}_{+}=\mathbb{R}\cup \{\infty \})$. We prove that w is the value of ${\Gamma }_{\infty }$, i.e., $w=v_{\infty }$.

Let $\pi$ be any strategy with finite support for the decision maker in ${\Gamma }_{\infty }$. Then it can be seen as a strategy for the decision maker in ${\Gamma }_n$ for some $n\in \mathbb{N}$. Now, we fix $n_0\in \mathbb{N}$ to be large enough. Now, considering the $n_0$-truncated SMDP ${\Gamma }_{n_0}$, we know by Sinha et al. (2017) [23] that the decision maker has an optimal pure semi-stationary strategy in ${\Gamma }_{n_0}$. Suppose $f_{n_0}^{*}$ is the optimal pure semi-stationary strategy of the decision maker in ${\Gamma }_{n_0}$. Now, by Theorem 1 of [23], we have $v_{n_0}={\varphi }_{n_0}\left(f_{n_0}^{*}\right)\ge {\varphi }_{n_0}\left(\pi \right)$ coordinate-wise, for all $\pi \in \Pi$. Since ${\left\{v_n\right\}}_{n=1}^{\infty }$ is an increasing sequence converging to $w\in {\mathbb{R}}_{+}^z$, we have $w\ge v_{n_0}\ge {\varphi }_{n_0}\left(\pi \right)$ for all $\pi \in \Pi$. Thus we have $w\ge v_{\infty }$. Now, take $ϵ>0$. As ${\left\{v_n\right\}}_{n=1}^{\infty }$ converges to w, there exists an $n_1\in \mathbb{N}$ such that $v_{n_1}\left(s\right)\ge w\left(s\right)-ϵ$ if $w\left(s\right)$ is finite and $v_{n_1}\left(s\right)\ge N$ if $w\left(s\right)=+\infty$. Now, considering the truncated SMDP ${\Gamma }_{n_1}$, we get a pure semi-stationary strategy $f_{n_1}^{*}$ of the decision maker, having a value vector $v_{n_1}$. Thus we have

$${\varphi }_{\infty }(s,f_{n_1}^{*})=v_{n_1}\left(s\right)\ge \left\{\begin{array}{cc}w\left(s\right)-ϵ\hfill & \hfill \mathrm{if}w\left(s\right)\in \mathbb{R}\\ N\hfill & \hfill \mathrm{if}w\left(s\right)=+\infty \end{array}\right.$$

(8)

The combination of the above arguments proves our theorem.    □

5. Algorithm to Compute Value and a Near-Optimal Pure Semi-Stationary Strategy in an Undiscounted SMDP with Countably Infinite Action Spaces

We discuss an algorithm to compute the value and a near-optimal pure semi-stationary strategy of the decision maker in an undiscounted SMDP with countably infinite action space.

5.1. The Truncation Algorithm

Algorithm 1 is the proposed algorithm, where we are considering truncated SMDP models here for the computation, thus the algorithm stops in finite time and is an efficient one for computational process. We follow the algorithm proposed by Mondal [24] to compute the optimal pure semi-stationary strategy of the decision maker and value vector in each truncated SMDP. We first present the algorithm with a practical, assumption-free stopping rule (Section 5.1), analyze its complexity and convergence (Section 5.2), and then give a certified stopping rule with a computable error guarantee (Section 5.3).  

Algorithm 1: Successive truncation algorithm (heuristic stopping rule)

Step 1: Fix $ϵ>0$ and a large positive integer $N\in \mathbb{N}$. Step 2: Given $n\in \mathbb{N}$, consider the truncated SMDPs ${\Gamma }_n$ and ${\Gamma }_{n+1}$. Step 3: Fix an initial state $s_0\in S$. Step 4: Apply the LP algorithm by Mondal ([24]) to compute optimal pure stationary strategies $f_n^{s_0*}$ and $f_{n+1}^{s_0*}$ for the initial state $s_0$ of the SMDPs ${\Gamma }_n$ and ${\Gamma }_{n+1}$ respectively and also the values $v_n\left(s_0\right)$ and $v_{n+1}\left(s_0\right)$, for given n. Step 5: if (i) $\mid v_{n+1}\left(s_0\right)-v_n\left(s_0\right)\mid <ϵ$ then $v_{n+1}\left(s_0\right)=v_{\infty }\left(s_0\right)$ and $f_{n+1}^{s_0*}$ is a near-optimal pure stationary strategy for the initial state $s_0$. do Repeat step 4 and $5\left(i\right)$ for all $s_0\in S=\{1,2,\dots ,z\}$ to get $f_{n+1}^{*}=\{f_{n+1}^{1*},f_{n+1}^{2*},\dots ,f_{n+1}^{s_0*},\dots ,f_{n+1}^{z*}\}$ as a near-optimal pure semi-stationary strategy. end do or else if (ii) $\mid v_{n+1}\left(s_0\right)\mid >N$ for all $s_0\in S$, then $v_{n+1}\left(s_0\right)=+\infty =v_{\infty }\left(s_0\right)$ and $f_{n+1}^{s_0*}$ is a near-optimal pure stationary strategy for the initial state $s_0$. do Repeat step 4 and $5\left(ii\right)$ for all $s_0\in S=\{1,2,\dots ,z\}$ to get $f_{n+1}^{*}=\{f_{n+1}^{1*},f_{n+1}^{2*},\dots ,f_{n+1}^{s_0*},\dots ,f_{n+1}^{z*}\}$ as a near-optimal pure semi-stationary strategy. end do or else if (iii) $\mid v_{n+1}\left(s_0\right)-v_n\left(s_0\right)\mid =0$ then $v_{n+1}\left(s_0\right)=v_{\infty }\left(s_0\right)$ and $f_{n+1}^{s_0*}$ is a pure stationary strategy for the initial state $s_0$. do Repeat step 4 and $5\left(iii\right)$ for all $s_0\in S=\{1,2,\dots ,z\}$ to get $f_{n+1}^{*}=\{f_{n+1}^{1*},f_{n+1}^{2*},\dots ,f_{n+1}^{s_0*},\dots ,f_{n+1}^{z*}\}$ as an optimal pure semi-stationary strategy. end do end if Step 6: Replace n by $n+1$ and return to step 3.

5.2. Complexity and Convergence

We consider here the truncated SMDP ${\Gamma }_n$, where the action set is

$$\{1,\dots ,n\}.$$

If we enlarge n, it will also enlarge the admissible finite support set. Thus, the truncated value $v_n$ is monotonically increasing in n, and

$$v_n\uparrow v_{\infty }\mathrm{as}n\to \infty .$$

We have shown in Theorem 2 that $v_{\infty }$ is the value of the SMDP.

Now, we consider the following two cases.

• Case I: If $v_{\infty }$ is finite.

Let us fix an initial state $s_0\in S$. Then, the sequence

$$\left\{v_n\left(s_0\right)\right\}$$

is monotone and converges to a finite limit. Hence, by Cauchy’s theorem, there exists $n_1\in \mathbb{N}$ such that

$$|v_{n+1}\left(s_0\right)-v_n\left(s_0\right)|<\epsilon ,\forall n\ge n_1.$$

Thus, Step 5(i) stops in finite time and returns

$$f_{n+1,}^{s_0*}$$

as an optimal pure stationary strategy for the initial state $s_0$.

• Case II: If $v_{\infty }$ is not finite.

If

$$v_{\infty }\left(s_0\right)=+\infty$$

then

$$v_n\left(s_0\right)\uparrow \infty .$$

Therefore, for an arbitrarily large $N\in \mathbb{N}$, we can find $n\in \mathbb{N}$ such that

$$v_{n+1}\left(s_0\right)>N.$$

Thus, Step 5(ii) stops and declares

$$v_{\infty }=(\infty ,\infty ,\dots ,\infty )$$

as the value of the infinite SMDP.

Hence, in both cases, the algorithmic criterion is satisfied for some finite $n\in \mathbb{N}$, and the algorithm terminates.

Repeating the argument for all initial states, we obtain the near-optimal pure semi-stationary strategy

$$f_{n+1}^{*}=\{f_{n+1}^{1*},f_{n+1}^{2*},\dots f_{n+1}^{z*}\}.$$

Now, we analyze the cost of producing a near-optimal pure semi-stationary strategy. The analysis separates the cost of a single linear program, the cost per truncation level, and the number of levels required for termination; it applies to both stopping rules of this section, since the per-level work is identical.

Proposition 1

(Per-iteration cost). Fix a truncation level n. For each initial state $s_0\in S$, the linear program solved in Step 4, in occupation-measure form, has $z·n$ decision variables and $z+1$ equality constraints. A path-following interior-point method solves it in $O\left(\sqrt{zn}L\right)$ iterations, where L is the bit length of the data, each iteration solving a normal-equation system in $O\left(z^3{n}^2\right)$ operations; the single-solve cost $L_{\mathrm{LP}}\left(zn\right)$ is polynomial in $zn$. Computing the value vector $v_n$ and an optimal pure semi-stationary strategy $f_{n+1}^{*}$ requires z such solves, at per-level cost $C_{\mathrm{level}}\left(n\right)=z{L}_{\mathrm{LP}}\left(zn\right)$.

Proof. 

The truncated SMDP ${\Gamma }_n$ has z states and at most n actions per state, and hence at most $z·n$ admissible state–action pairs, which index the occupation-measure variables; the equality constraints are the z balance constraints together with one normalization constraint. By Mondal [24], for a fixed initial state the value and an optimal pure stationary strategy of a finite multichain SMDP are recovered from an optimal solution of this program; repeating over the z initial states assembles $f_{n+1}^{*}$. The interior-point bounds are classical: $O\left(\sqrt{v}L\right)$ iterations [25] for v variables, each dominated by an $O\left(c{v}^2\right)$ normal-equation solve for c constraints, giving the cubic single-solve bound $O\left({\left(zn\right)}^{3}L\right)$ [26]. Multiplying by the z initial states, we get $C_{\mathrm{level}}\left(n\right)$. □

Remark 1.

Since $A_n\left(s\right)\subset A_{n+1}\left(s\right)$, the optimal basis at level n warm-starts the program at level $n+1$, so each re-optimization typically reduces to a few dual-simplex pivots. This does not change the worst-case bound but accounts for the favorable observed running times.

Let $n^{*}$ be the truncation level at which the stopping rule first fires. By Proposition 1, the total cost is

$$C_{\mathrm{total}}={\displaystyle \sum _{n=1}^{n^{*}}}z{L}_{\mathrm{LP}}\left(zn\right)=O\left(z{n}^{*}{L}_{\mathrm{LP}}\left(z{n}^{*}\right)\right),$$

(9)

polynomial in z and $n^{*}$ once $n^{*}$ is fixed. The level $n^{*}$ is finite but is not bounded by the input size in general: it depends on the accuracy parameters $(ϵ,N)$ and on the rate of convergence of the monotone sequence $v_n\uparrow v_{\infty }$ (Theorem 2). When $v_{\infty }$ is finite, $n^{*}=n^{*}\left(ϵ\right)$ is governed by the convergence rate; when $v_{\infty }\left(s\right)=+\infty$ for all $s\in S$, $n^{*}=n^{*}\left(N\right)$ is the first level exceeding the cutoff N. In both cases monotonicity guarantees termination after finitely many iterations. The algorithm is thus polynomial per iteration with a finite termination level that depends on the accuracy parameters and on the model rather than on the number of states; it is not strongly polynomial, and we make no such claim.

Numerical Example

Consider an undiscounted infinite SMDP ${\Gamma }_{\infty }$ with three states $S=\{1,2,3\}$, $A\left(1\right)=A\left(2\right)=A\left(3\right)=\{1,2,\cdots \}$. Thus, in all states, the decision maker has action sets whose cardinality is not finite. The rewards, probability of transition, and the mean (expected) sojourn times for the decision maker are shown below in each cell of the infinite SMDP:

	1 $(1,0,0)$ 1		1 $(1,0,0)$ 2.1		1 $({\displaystyle \frac{1}{2}},{\displaystyle \frac{1}{4}},{\displaystyle \frac{1}{4}})$ 3.4
	$2^{0.86}$ $(1,0,0)$ 1		$(1+{\displaystyle \frac{1}{4}})$ $({\displaystyle \frac{1}{2}},0,{\displaystyle \frac{1}{2}})$ 2.1		$(1+{\displaystyle \frac{1}{8}})$ $({\displaystyle \frac{3}{4}},{\displaystyle \frac{1}{8}},{\displaystyle \frac{1}{8}})$ 3.4
State-1:	$3^{0.86}$ $(1,0,0)$ 1	State-2:	$(1+{\displaystyle \frac{1}{4}}+{\displaystyle \frac{1}{9}})$ $({\displaystyle \frac{1}{3}},0,{\displaystyle \frac{2}{3}})$ 2.1	State-3:	$(1+{\displaystyle \frac{1}{8}}+{\displaystyle \frac{1}{27}})$ $({\displaystyle \frac{5}{6}},{\displaystyle \frac{1}{12}},{\displaystyle \frac{1}{12}})$ 3.4
	⋮		⋮		⋮
	$n^{0.86}$ $(1,0,0)$ 1		$(1+{\displaystyle \frac{1}{4}}+\dots {\displaystyle \frac{1}{n^2}})$ $({\displaystyle \frac{1}{n}},0,1-{\displaystyle \frac{1}{n}})$ 2.1		$(1+{\displaystyle \frac{1}{8}}+\dots {\displaystyle \frac{1}{n^3}})$ $(1-{\displaystyle \frac{1}{2n}},{\displaystyle \frac{1}{4n}},{\displaystyle \frac{1}{4n}})$ 3.4
	⋮		⋮		⋮

We can write the reward function as:

$r(1,n)=n^{0.86}$, $r(2,n)={\sum }_{n=1}^{\infty }(1/n^2)$ and $r(3,n)={\sum }_{n=1}^{\infty }(1/n^3)$ for $n\in \mathbb{N}$.

The transition probabilities can be written as:

$p(1\mid 1,n)=1$, $p(2\mid 1,n)=0$, $p(3\mid 1,n)=0.$

$p(1\mid 2,n)={\displaystyle \frac{1}{n}}$, $p(2\mid 2,n)=0$, $p(3\mid 2,n)=1-{\displaystyle \frac{1}{n}}.$

$p(1\mid 3,n)=1-{\displaystyle \frac{1}{2n}}$, $p(2\mid 3,n)={\displaystyle \frac{1}{4n}}$, $p(3\mid 3,n)={\displaystyle \frac{1}{4n}}.$

The mean sojourn times are given as:

$\overline{\tau }(1,n)=1$, $\overline{\tau }(2,n)=2.1$, $\overline{\tau }(3,n)=3.4$.

For each $n\in \mathbb{N}$, we write the linear programming problem with respect to the variables $x=(x_{1n},x_{2n},x_{3n}),y=(y_{1n},y_{2n},y_{3n})$ and t for this model as:

$$max[{\displaystyle \sum _{i=1}^n}{i}^{0.86}{x}_{1i}+{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{i^2}}{x}_{2i}+{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{i^3}}{x}_{3i}]$$

(10)

With respect to the constraints:

(i)

${\sum }_{i=1}^n[-{\displaystyle \frac{1}{i}}{x}_{2i}+({\displaystyle \frac{1}{2i}}-1)x_{3i}]=0$

(ii)

${\sum }_{i=1}^n[x_{2i}-{\displaystyle \frac{1}{4i}}{x}_{3i}]=0$

(iii)

${\sum }_{i=1}^n[({\displaystyle \frac{1}{i}}-1)x_{2i}+(1-{\displaystyle \frac{1}{4i}})x_{3i}]=0$

(iv)

${\sum }_{i=1}^n[x_{1i}-{\displaystyle \frac{1}{i}}{y}_{2i}+({\displaystyle \frac{1}{2i}}-1)y_{3i}]-{\delta }_{s_{0}1}t=0$

(v)

${\sum }_{i=1}^n[x_{2i}+y_{2i}-{\displaystyle \frac{1}{4i}}{y}_{3i}]-{\delta }_{s_{0}2}t=0$

(vi)

${\sum }_{i=1}^n[x_{3i}+({\displaystyle \frac{1}{i}}-1)y_{2i}+(1-{\displaystyle \frac{1}{4i}})y_{3i}]-{\delta }_{s_{0}3}t=0$

(vii)

${\sum }_{i=1}^n[x_{1i}+2.1x_{2i}+3.4x_{3i}]=1$

where $x_{1n},x_{2n},x_{3n},y_{1n},y_{2n},y_{3n},t\ge 0$ for all $n\in \mathbb{N}$. Let us fix $N=100$ in this case. The values of $v_{n+1}$ and $v_n$ for each $n\in \mathbb{N}$ are given below.

Value of n	$v_n$	$v_{n+1}$
1	$(1,1,1)$	$(1.828427,1.828427,1.828427)$
2	$(1.828427,1.828427,1.828427)$	$(2.196152,2.196152,2.196152)$
3	$(2.196152,2.196152,2.196152)$	$(3,3,3)$
4	$(3,3,3)$	$(4.180340,4.180340,4.180340)$
5	$(4.180340,4.180340,4.180340)$	$(6.696938,6.696938,6.696938)$
6	$(6.696938,6.696938,6.696938)$	$(8.520259,8.520259,8.520259)$
7	$(8.520259,8.520259,8.520259)$	$(12.627417,12.627417,12.627417)$
8	$(12.627417,12.627417,12.627417)$	$(17,17,17)$
9	$(17,17,17)$	$(23.622777,23.622777,23.622777)$
10	$(23.622777,23.622777,23.622777)$	$(30.482873,30.482873,30.482873)$
11	$(30.482873,30.482873,30.482873)$	$(37.569219,37.569219,37.569219)$
12	$(37.569219,37.569219,37.569219)$	$(45.872167,45.872167,45.872167)$
13	$(45.872167,45.872167,45.872167)$	$(53.383203,53.383203,53.383203)$
14	$(53.383203,53.383203,53.383203)$	$(59.094750,59.094750,59.094750)$
15	$(59.094750,59.094750,59.094750)$	$(67,67,67)$
16	$(67,67,67)$	$(77.092796,77.092796,77.092796)$
17	$(77.092796,77.092796,77.092796)$	$(85.367532,85.367532,85.367532)$
18	$(85.367532,85.367532,85.367532)$	$(92.819080,92.819080,92.819080)$
19	$(92.819080,92.819080,92.819080)$	$(99.442719,99.442719,99.442719)$
20	$(99.442719,99.442719,99.442719)$	$(108.234090,108.234090,108.234090)$
⋮	⋮	⋮
1000	$(1851.73,1851.73,1851.73)$	$(1866.5,1866.5,1866.5)$

Now, applying our algorithm, we conclude $v_{\infty }=(+\infty ,+\infty ,+\infty )$. Now, for the initial state 1, the optimal pure stationary strategy is $f_1^{*}=(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0)$. Similarly, for the initial state 2 and 3, the optimal pure stationary strategies are $f_2^{*}=(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1)$ and $f_3^{*}=(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1)$. Thus, $f^{*}=(f_1^{*},f_2^{*},f_3^{*})$ is the pure near-optimal semi-stationary strategy. Intuitively,

$$v^{\infty }=(+\infty ,+\infty ,+\infty )$$

reflects that the reward

$$r(i,n)=n^{0.86}$$

is unbounded for $n\in \mathbb{N}$.

Here, the sojourn time

$$\overline{\tau }(i,n)=1$$

is fixed for state 1. Thus, the decision maker can make the long-run limiting ratio average reward as large as desired by committing to actions of sufficiently high index.

No single stationary (pure) or pure semi-stationary strategy can attain the supremum. Thus, we fail to obtain an optimal strategy. Therefore, for each N, we set a tolerance $ϵ>0$ and obtain an $ϵN$-optimal (near-optimal) pure semi-stationary strategy.

5.3. Certified Truncation Error and Corrected Stopping Rule

The stopping test $|v_{n+1}\left(s\right)-v_n\left(s\right)|<ϵ$ compares two successive truncations and does not by itself certify closeness of $v_n$ to the infinite-action value $v_{\infty }$. We replace it with a rule based on the Bellman residual of the truncated solution, which bounds the remaining error directly, and we delimit precisely the regime in which such a certificate exists. We work in the data-transformed average-reward model of Appendix A (unit sojourn times, reward $\widehat{r}$, transitions $\widehat{q}$, optimal gain $g^{*}=v_{\infty }$ which is state-independent under (C1), optimal relative-value function $h^{*}$). In addition to the standing finiteness of S, we assume:

5.4. Assumptions

Assumption 1

(Bounded normalized reward). $|\widehat{r}(s,a)|=|r(s,a)/\overline{\tau }(s,a)|\le M$ for all $(s,a)$, for some $M<\infty$.

Assumption 2

(Ergodicity). There exists a distinguished state $x_0\in S$ and a finite T such that ${\mathbb{E}}_f[T_{x_0}\mid X_0=y]\le T$ for every $y\in S$ and every finite-support stationary strategy f, where $T_{x_0}=inf\{n\ge 1:X_n=x_0\}$.

Assumption 3

(Tail regularity). For each $s\in S$, the normalized reward $\widehat{r}(s,a)=r(s,a)/\overline{\tau }(s,a)$ is eventually non-increasing in the action index a; equivalently, ${sup}_{a>n}\widehat{r}(s,a)$ is a computable function of n that converges to a finite limit as $n\to \infty$.

Assumptions 1 and 2 are those already used for the optimality equation of Appendix A. Assumption 3 is the additional structural hypothesis that makes the excluded actions’ best possible contribution computable; it holds, for instance, whenever rewards are bounded and the per-action reward does not grow with the action index. It necessarily fails in the regime of the example of Section Numerical Example (Level 3 in Section 5.2), where $r(1,a)=a^{0.86}$ is increasing and $v_{\infty }=+\infty$ (see Remark 2).

5.5. The Certified Error Bound

Let $(v_n,h_n)$ be the gain and relative-value function returned by the truncated problem ${\Gamma }_n$, i.e., the solution of the truncated optimality equation

$$h_n\left(s\right)=\underset{a\in A_n\left(s\right)}{max}\left[\widehat{r}(s,a)-v_n+\sum _{s^{\prime }}\widehat{q}(s^{\prime }\mid s,a)h_n\left(s^{\prime }\right)\right],s\in S.$$

(11)

Define the full-action Bellman residual of the truncated solution by

$${\rho }_n\left(s\right)=\underset{a\in A\left(s\right)}{max}\left[\widehat{r}(s,a)-v_n+\sum _{s^{\prime }}\widehat{q}(s^{\prime }\mid s,a)h_n\left(s^{\prime }\right)\right]-h_n\left(s\right),{\rho }_n=\underset{s\in S}{max}{\rho }_n\left(s\right).$$

(12)

Proposition 2

(Certified truncation error). Under Assumptions 1–3, the residual ${\rho }_n$ in (12) satisfies ${\rho }_n\ge 0$, is computable from finite data, and bounds the truncation error:

$$0\le v_{\infty }-v_n\le T{\rho }_n,$$

(13)

and ${\rho }_n\to 0$ as $n\to \infty$. Consequently, for any $ϵ>0$, once $T{\rho }_n<ϵ$, the truncated strategy $f_n^{*}$ is a certified ϵ-optimal pure semi-stationary strategy: ${\varphi }_{\infty }(s,f_n^{*})\ge v_{\infty }\left(s\right)-ϵ$ for all $s\in S$.

Proof. 

Since $A_n\left(s\right)\subseteq A\left(s\right)$ and the maximum over $A_n\left(s\right)$ equals $h_n\left(s\right)$ by (11), the maximum over the larger set $A\left(s\right)$ is at least $h_n\left(s\right)$; hence ${\rho }_n\left(s\right)\ge 0$.

Splitting the maximum in (12) into $a\le n$ and $a>n$ and using (11) for the former,

$${\rho }_n\left(s\right)=max\left\{0,\underset{a>n}{sup}\left[\widehat{r}(s,a)-v_n+\sum _{s^{\prime }}\widehat{q}(s^{\prime }\mid s,a)h_n\left(s^{\prime }\right)\right]-h_n\left(s\right)\right\}.$$

(14)

Here, we bound ${\rho }_n$ above by a quantity that involves only the included data and the single next excluded action, thereby avoiding any evaluation of the infinite supremum in (14). Under Assumption 3, the tail $a\mapsto \widehat{r}(s,a)$ is non-increasing for $a>n$ (for n beyond the index at which monotonicity sets in), so ${sup}_{a>n}\widehat{r}(s,a)=\widehat{r}(s,n+1)$. Bounding the transition term in (14) by ${\sum }_{s^{\prime }}\widehat{q}(s^{\prime }\mid s,a)h_n\left(s^{\prime }\right)\le {max}_{s^{\prime }}{h}_n\left(s^{\prime }\right)$ uniformly in a, we obtain the finite, computable upper estimate

$${\widehat{\rho }}_n\left(s\right)=\left[\widehat{r}(s,n+1)-v_n+\underset{s^{\prime }}{max}{h}_n\left(s^{\prime }\right)\right]-\underset{s\prime }{min}{h}_n(s\prime ),0\le {\rho }_n\le \underset{s}{max}{\widehat{\rho }}_n\left(s\right)=:{\widehat{\rho }}_n.$$

(15)

Thus, ${\widehat{\rho }}_n$ is evaluated from the truncated solution $(v_n,h_n)$ and the reward of action $n+1$ alone.

Now, it suffices to show ${\widehat{\rho }}_n\to 0$. We use that the relative values converge. By Assumption 2, for every truncation level, the relative-value function admits the first-passage representation $h_n\left(s\right)={\mathbb{E}}_{f_n^{*}}\left[{\sum }_{k=0}^{T_{x_0}-1}(\widehat{r}(X_k,A_k)-v_n)\mid X_0=s\right]$ with $h_n\left(x_0\right)=0$, so by Assumption 1 the span is uniformly bounded, $sp\left(h_n\right)={max}_s{h}_n\left(s\right)-{min}_s{h}_n\left(s\right)\le 2MT$ for all n. Together with the monotone convergence $v_n\uparrow v_{\infty }$ of Theorem 2, this uniform bound yields, along the truncation sequence, $v_n\to v_{\infty }$ and (anchoring $h_n\left(x_0\right)=0$) $h_n\to h^{*}$ coordinate-wise, where $(v_{\infty },h^{*})$ solves the full optimality equation. Passing to the limit in (15), and using $\widehat{r}(s,n+1)\to {\widehat{r}}_{\infty }\left(s\right):={lim}_{a\to \infty }\widehat{r}(s,a)$ from Assumption 3,

$$\underset{n\to \infty }{lim\; sup}{\widehat{\rho }}_n\le \underset{s}{max}\left[{\widehat{r}}_{\infty }\left(s\right)-v_{\infty }+\underset{s^{\prime }}{max}{h}^{*}\left(s^{\prime }\right)-h^{*}\left(s\right)\right].$$

(16)

Since $\widehat{r}(s,a)$ is non-increasing in a, ${\widehat{r}}_{\infty }\left(s\right)\le \widehat{r}(s,a)$ for every finite a, in particular, the limiting tail reward cannot exceed the reward of any action retained at the optimum. Evaluating the full optimality equation at the optimal action shows $\widehat{r}(s,a^{*})-v_{\infty }+{\sum }_{s^{\prime }}\widehat{q}(s^{\prime }\mid s,a^{*})h^{*}\left(s^{\prime }\right)=h^{*}\left(s\right)$, so the bracket in (16) is non-positive for each s (the excluded tail can do no better than the optimal included action). Combined with ${\rho }_n\ge 0$ this gives ${\widehat{\rho }}_n\to 0$, hence ${\rho }_n\to 0$.

Error bound. From (12), for every $a\in A\left(s\right)$,

$$\widehat{r}(s,a)-v_n+\sum _{s^{\prime }}\widehat{q}(s^{\prime }\mid s,a)h_n\left(s^{\prime }\right)\le h_n\left(s\right)+{\rho }_n.$$

(17)

Let $a^{*}\left(s\right)$ attain the maximum in the full optimality equation $h^{*}\left(s\right)=\widehat{r}(s,a^{*})-v_{\infty }+{\sum }_{s^{\prime }}\widehat{q}(s^{\prime }\mid s,a^{*})h^{*}\left(s^{\prime }\right)$. Writing $d=h^{*}-h_n$ and evaluating (17) at $a=a^{*}\left(s\right)$, then subtracting it from the full optimality equation, gives

$$v_{\infty }-v_n\le {\rho }_n+\left(P^{a^{*}}d\right)\left(s\right)-d\left(s\right),$$

(18)

where $P^{a^{*}}$ is the transition matrix of the $v_{\infty }$-optimal policy. Iterating (18) along the chain $P^{a^{*}}$ until the first passage to $x_0$, the telescoping term $(P^{a^{*}}d-d)$ accumulates to $d\left(x_0\right)-d\left(s\right)$ plus a remainder of at most one factor ${\rho }_n$ per step; by Assumption A2, the expected number of steps is at most T, while Assumption A1 guarantees d is bounded so the telescoping is legitimate. Taking expectations yields $v_{\infty }-v_n\le T{\rho }_n$. Combining with ${\rho }_n\ge 0$ and $v_n\le v_{\infty }$ (monotonicity, Theorem 2) gives (13). The $ϵ$-optimality of $f_n^{*}$ when $T{\rho }_n<ϵ$ is immediate. □

The multiplicative constant is exactly the first-passage bound T of Assumption 2; the reward bound M enters only as the precondition that $h_n$ and $h^{*}$ have finite span, not in the constant.

5.6. Corrected Algorithm

Steps 1–3 of the algorithm of Section 5 are unchanged. Steps 4 and 5 are replaced as follows; the modification is to compute the residual ${\rho }_n$ and to stop on the certified bound $T{\rho }_n$ rather than on the one-step increment.

Step 4.

For the current truncation level n and each initial state $s_0\in S$, solve the linear program for ${\Gamma }_n$ to obtain the value $v_n\left(s_0\right)$, the optimal pure stationary action, and the relative-value function $h_n$. Assemble $f_n^{*}=(f_n^{1*},\dots ,f_n^{z*})$.

Step 5.

Compute the residual estimate ${\widehat{\rho }}_n=\left\{[\widehat{r}(s,n+1)-v_n+\parallel h_n{\parallel }_{\infty }]-{min}_{s^{\prime }}{h}_n\left(s^{\prime }\right)\right\}$ from (15), and the certified error $c_n=T{\widehat{\rho }}_n$. Then:

(i)

if$c_n<ϵ$: stop; $f_n^{*}$ is a certified $ϵ$-optimal pure semi-stationary strategy, with $v_{\infty }-v_n\le c_n<ϵ$;

(ii)

else if$v_n\left(s\right)>N$ for all s (rewards unbounded, Assumption A3 fails): stop and certify $v_{\infty }\left(s\right)=+\infty$ to resolution N; $f_n^{*}$ is N-optimal;

(iii)

else if${\widehat{\rho }}_n=0$: stop; $f_n^{*}$ is exactly optimal ($v_n=v_{\infty }$);

(iv)

else replace n by $n+1$ and return to Step 4.

Relative to the original algorithm, branch (i) stops on a certified bound on the actual error rather than on the one-step change; branch (ii) is the unchanged $+\infty$-cutoff; branch (iii) is the exact-optimality certificate. The increment test $|v_{n+1}-v_n|<ϵ$ is removed.

Remark 2

(Scope). The certificate exists precisely under Assumptions 1–3. When rewards are unbounded and Assumption 3 fails—as in the example of Section Numerical Example (Level 3 in Section 5.2), where $r(1,a)=a^{0.86}$ grows and $v_{\infty }=+\infty$—no finite a priori or computable truncation-error bound can exist, since $v_n\to +\infty$. There, the algorithm certifies divergence via branch (ii) rather than claiming closeness to a finite optimum. This yields a sharp characterization: a certified truncation-error bound is available in the bounded-reward, ergodic, tail-regular sub-class and is provably unavailable outside it. The one-step increment test, by contrast, gives no guarantee in either regime.

Numerical Example

The previous worked example illustrates the algorithm and the case $v_{\infty }=\infty$, but a single hand-constructed instance is not sufficient to assess how the method behaves as the problem grows or across varied problem data. To this end, we evaluate the algorithm on randomly generated SMDPs. An instance with z states and n actions per state is specified by a seed and the following sampling scheme: each expected sojourn time $\overline{\tau }(s,a)$ is drawn uniformly from $[0.5,3.0]$ (i.e., bounded away from zero). Here, each transition distribution $q(·\mid s,a)$ is obtained by drawing independent uniform weights, adding a small positive floor, and normalizing, which keeps the induced chains communicating. Each reward $r(s,a)$ is drawn uniformly from $[0,1]$ in the bounded-reward regime, or set to $n^{0.86}$ (for some $n\in \mathbb{N}$) times a uniform factor in the unbounded regime that mirrors the worked example. Fixing the seed makes every reported result reproducible.

Table 1 displays one such case with $z=4$ states, truncated to its first five actions per state for readability; the transition, reward, and sojourn-time data are shown for each state-action pair. In the experiments that follow, the action sets are truncated at substantially larger levels and reported quantities are averaged over several seeds. We use these instances to study the algorithm’s running time and memory as a function of the number of states and to evaluate the certified truncation-error bound of Proposition 2.

Table 1. A randomly generated undiscounted SMDP instance with $z=4$ states, shown truncated to the first 5 actions per state (bounded-reward regime, seed 11). For each state s and action a, the table lists the transition probabilities $q(s^{\prime }\mid s,a)$ over the next states $s^{\prime }\in \{1,2,3,4\}$, the reward $r(s,a)$, and the expected sojourn time $\overline{\tau }(s,a)$. Each transition row sums to one. In the experiments, the action sets are truncated at much larger levels; five actions are displayed here only for readability.

State	a	$\mathit{q}(1\mid \mathit{s},\mathit{a})$	$\mathit{q}(2\mid \mathit{s},\mathit{a})$	$\mathit{q}(3\mid \mathit{s},\mathit{a})$	$\mathit{q}(4\mid \mathit{s},\mathit{a})$	$\mathit{r}(\mathit{s},\mathit{a})$	$\overline{\mathit{\tau }}(\mathit{s},\mathit{a})$
1	1	0.426	0.105	0.249	0.220	0.144	0.821
	2	0.185	0.294	0.131	0.390	0.443	1.748
	3	0.473	0.092	0.266	0.169	0.786	2.004
	4	0.076	0.538	0.273	0.113	0.895	0.572
	5	0.330	0.115	0.434	0.122	0.759	0.870
2	1	0.067	0.201	0.317	0.416	0.035	2.821
	2	0.443	0.346	0.180	0.031	0.359	0.676
	3	0.084	0.417	0.203	0.296	0.163	0.824
	4	0.061	0.049	0.520	0.370	0.999	2.871
	5	0.330	0.338	0.128	0.205	0.144	2.055
3	1	0.045	0.537	0.312	0.107	0.244	1.422
	2	0.440	0.137	0.062	0.361	0.357	1.778
	3	0.447	0.036	0.404	0.113	0.061	2.157
	4	0.107	0.051	0.258	0.584	0.870	1.188
	5	0.261	0.434	0.129	0.176	0.636	0.845
4	1	0.113	0.378	0.365	0.144	0.160	2.470
	2	0.280	0.210	0.459	0.052	0.498	2.176
	3	0.067	0.258	0.167	0.508	0.079	1.781
	4	0.279	0.210	0.250	0.261	0.611	2.542
	5	0.310	0.364	0.074	0.252	0.232	1.873

The resulting transformation parameter, the limiting-ratio-average value, and the dimensions of the linear program for this instance are summarised in Table 2, and the convergence of the truncated value $v_n$ with the truncation level n is reported in Table 3.

Table 2. Outcome of the randomly generated instance of Table 1 (bounded-reward regime, seed 11). Because this instance is communicating, the value vector $v=\left(v\right(1),\dots ,v(4\left)\right)$ has identical components; we therefore report the common value $v\left(s\right)$ as a scalar. In general, the method computes the full value vector in ${\mathbb{R}}_{+}^z$, one component per initial state, as in the worked example of Section Numerical Example (Level 3 in Section 5.2) where $v_{\infty }=(+\infty ,+\infty ,+\infty )$.

Quantity	Value
Number of states z	4
Actions per state (truncation level n)	5
Transformation parameter $\eta$	$0.6324$
Limiting-ratio-average value v	$0.89164$
Optimal pure stationary action per state	$(5,5,4,3)$
LP decision variables ($z·n$)	20
LP equality constraints ($z+1$)	5

Table 3. Convergence of the truncated value $v_n$ with the truncation level n for the instance of Table 1. As above, $v_n\left(s\right)$ is state-independent for this communicating instance and is reported as a scalar. The sequence is non-decreasing in n (Theorem 2) and stabilises once the admitted actions no longer improve the optimum, at which point $v_n\left(s\right)=v\left(s\right)$ to the displayed precision.

Truncation Level n	Value ${\mathit{v}}_{\mathit{n}}$	Optimal Action Per State
1	$0.29658$	$[1,1,1,1]$
2	$0.29658$	$[1,1,1,1]$
3	$0.52342$	$[1,1,1,3]$
4	$0.55541$	$[1,1,4,3]$
5	$0.89164$	$[5,5,4,3]$

Formulating in MATLAB, we get the following figures. Figure 1 plots the truncated value $v_n$ against the truncation level n for both the bounded and unbounded-reward regimes. Figure 2 reports the total running time and the termination level $n^{*}$ as functions of the number of states z. Figure 3 compares the one-step increment $|v_{n+1}-v_n|$ with the true truncation gap $v_{\infty }-v_n$, which shows that the increment is a convergence indicator rather than a certificate of closeness to $v_{\infty }$. Finally, Figure 4 shows that the certified bound $c_n=T{\rho }_n$ of Proposition 2 dominates the true gap at every level and decreases to zero.

Figure 1. Truncated value $v_n$ versus truncation level n, computed in MATLAB. (a) Bounded rewards: $v_n$ increases to a finite limit $v_{\infty }$. (b) Unbounded rewards ($r\propto a^{0.86}$): $v_n$ increases without bound, so $v_{\infty }=+\infty$, mirroring the worked example of Section Numerical Example (Level 3 in Section 5.2).

Figure 2. Cost of the algorithm versus the number of states z (means over several seeds), computed in MATLAB. (a) Total time to meet the stopping rule, on a logarithmic scale. (b) Termination level $n^{*}$. The growth in both quantities reflects the $O(z·n)$ size of each linear program together with the increase of $n^{*}$ with z.

Figure 3. True truncation gap $v_{\infty }-v_n$ and the one-step increment $|v_{n+1}-v_n|$ versus n (logarithmic scale), computed in MATLAB on a bounded-reward instance. The increment falls to near zero at levels where the true gap is still appreciable, showing that the one-step test alone is a convergence indicator rather than a certificate of closeness to $v_{\infty }$.

Figure 4. Certified error bound $c_n=T{\rho }_n$ of Proposition 2, the true gap $v_{\infty }-v_n$, and the old one-step increment, versus n (logarithmic scale), computed in MATLAB. The certified bound dominates the true gap at every level and decreases towards zero, so stopping when $c_n<ϵ$ guarantees $v_{\infty }-v_n<ϵ$.

Remark 3

(The two stopping rules). We retain both stopping rules because they occupy different points on the trade-off between computational economy and rigor, and neither dominates the other across all instances.

The heuristic rule of Section 5.1, based on the one-step change $|v_{n+1}\left(s\right)-v_n\left(s\right)|<ϵ$, requires no structural assumptions beyond those of the model, no ergodicity constant, and no auxiliary computation; it is therefore applicable to every instance, including those with unbounded rewards and $v_{\infty }=+\infty$. We stress, however, that this test is an indicator of convergence and not a certificate of closeness to $v_{\infty }$: as shown in Figure 3, the increment may fall below ϵ at truncation levels where the true gap $v_{\infty }-v_n$ remains appreciable.

The certified rule of Section 5.6, based on the Bellman residual ${\rho }_n$ (Proposition 2), provides the rigorous computable guarantee $0\le v_{\infty }-v_n\le T{\rho }_n$ and hence a certified ϵ-optimal strategy when it stops (Figure 4). This guarantee is obtained at the cost of computing the relative-value function $h_n$ and the residual ${\rho }_n$ at each truncation level, and of the additional hypotheses-bounded rewards, ergodicity, and tail regularity (Assumptions 1–3) under which the certificate is valid.

The two rules are thus complementary. The heuristic rule is the natural choice for rapid exploration, for problems in which the certifying assumptions cannot be verified, and for the regime $v_{\infty }=+\infty$, where no finite truncation-error bound can exist and only the heuristic together with the N-cutoff branch is available (Remark 2). The certified rule is used whenever a guaranteed accuracy is required and its assumptions hold; in that case it is the preferred rule, since it stops on a bound on the actual error rather than on the one-step change. Both rules share the same per-level linear-programming work, so the complexity analysis of Section 5.2 applies to each; the certified rule incurs only the additional, polynomially-bounded cost of forming $h_n$ and ${\rho }_n$.

6. Conclusions

We have studied undiscounted (limiting ratio average) semi-Markov decision processes with finite state space and countably infinite action space, imposing no restriction on the reward function. For this class, we proved the existence of the value and of a near-optimal pure semi-stationary strategy for the decision maker, and we proposed a truncation-based algorithm that computes the value and such a strategy in finite time, solving one finite linear program per initial state at each truncation level.

We complemented the existence result with a quantitative analysis of the algorithm. We showed that the per-level cost is polynomial in the number of state–action pairs, while the number of truncation levels needed for termination is finite but governed by the accuracy parameters and the model rather than by the number of states; the algorithm is therefore efficient per iteration but not strongly polynomial. We presented two stopping rules: a practical, assumption-free heuristic based on the one-step change in value, applicable to every instance including those with infinite value; and a certified rule based on the Bellman residual of the truncated solution, which provides a computable error bound guarantee $0\le v_{\infty }-v_n\le T{\rho }_n$ on the truncation error under bounded rewards together with standard ergodicity and tail-regularity conditions. We showed that such a certificate is available precisely in this sub-class and is provably unavailable when the value is infinite. We also derived an optimality equation for these models under ergodicity conditions, and we reported numerical experiments, including randomly generated instances, illustrating the algorithm and its scalability.

Several directions remain open. It would be of interest to weaken the tail-regularity condition under which the certified error bound holds, and to develop approximate or learning-based methods that scale to large state spaces, where the exact linear-programming approach becomes expensive. Another natural direction is the undiscounted zero-sum two-person perfect-information semi-Markov game in which a non-fixed player has countably infinite actions in each state; the existence of near-optimal pure semi-stationary strategies in such games is worth studying.