Structural Properties and a Revised Value Iteration Algorithm for Dynamic Capacity Expansion and Reduction

Abstract

This manuscript introduces a generalized Markov Decision Process (MDP) model for dynamic capacity planning in the presence of stochastic time-nonhomogeneous demand, wherein system capacity may be flexibly increased or decreased throughout a finite planning horizon. The model includes investment, disinvestment, maintenance, operational, and shortage costs, in addition to a salvage value at the end of the planning horizon. Under very realistic conditions, we investigate the structural properties of the optimal policy and demonstrate its monotonic structure. By leveraging these properties, we propose a revised value iteration algorithm that capitalizes on the intrinsic structure of the problem, thereby achieving enhanced computational efficiency compared to traditional dynamic programming techniques. The proposed model is applicable across a range of sectors, including manufacturing systems, cloud-computing services, logistics systems, healthcare resource management, power capacity planning, and other intelligent infrastructures driven by Industry 4.0.

1. Introduction

The discipline of capacity planning is distinguished by a substantial body of academic research, encompassing a wide array of application domains, including (a) power generation and distribution; (b) production and supply chain logistics; (c) network systems such as electricity, radio, and mobile communications; and (d) hospital capacity planning. For an overview of this topic, the reader is referred to Ref. [1] along with the review articles listed therein.

There exists a broad array of modelling and solution methodologies for capacity planning, illustrating the heterogeneity and complexity inherent in real-world scenarios. This includes, but is not limited to, the following.

A diverse array of modelling techniques are currently used for capacity planning, including (a) mathematical programming techniques [2], (b) stochastic and dynamic programming methods [3,4], (c) robust optimization [5], and (d) simulation-based optimization [6,7]. With the exception of stochastic dynamic programming methods, these techniques are normally applied to deterministic problems or employed in policy evaluation. The challenge of sequential decision-making under stochastic demand conditions is most effectively modelled via stochastic dynamic programming. AlDurgam et al. [1] confirmed that the topic of sequential decision-making under conditions of uncertainty is considerably underrepresented in the literature in comparison to the aforementioned methodologies. A Markov decision process (MDP) is a discrete stochastic dynamic programming technique that is extensively employed to model sequential decision-making under uncertainty scenarios frequently observed in capacity planning. The assumption of static demand can result in significant ramifications, including, as thoroughly documented, instances of capacity shortages or excess capacities [8].

Characterising the optimal policies in MDPs facilitates the practical application of intricate models by resulting in simple decision rules [9], such as threshold or monotone policies, which can be easily executed by decision makers. Furthermore, these characteristics reduce the computational complexity of MDPs by limiting the solution space [10,11]. Recently Krishnamurthy [12] demonstrated that in Markov decision processes, optimal policies increase with the state under broader conditions than supermodularity, such as sigmoidal rewards and non-supermodular transitions. Their research confirmed monotonicity with totally positive order-three transition matrices and concave value functions. Lee et al. [13] analysed monotone policy iteration in Markov decision processes, offering convergence and optimality conditions and a modified policy iteration algorithm. They assessed nineteen state-ordering rules’ effects, finding a time–quality trade-off: faster orderings excel in structured scenarios, while random orderings reduce the optimality gap with more of a computational cost.

Martínez-Costa et al. [14] observed that strategic capacity-planning models generally emphasize expansion while paying less attention to capacity reduction—a gap that is particularly salient in global markets with rapid product cycles. In practice, workforce and logistics planning require the ability to scale resources both up and down. This pattern extends to the MDP literature: several studies (

Table 1. Summary of the most relevant MDB-based research articles.

Study	Capacity Expansion vs. Reduction	Horizon	Uncertainties
[15]	Expansion	Finite	Product price, demand (Poisson), and lifecycle
[11]	Expansion	Finite	Demand, price, and yield uncertainties
[16]	Expansion	Finite	Markovian demand
[17]	Expansion and reduction	2-period	Demand linear in price with normal noise
[18]	Expansion	Finite	Stochastic returns (Binomial pdf)
[19]	Expansion	Finite	General stochastic demand (can be time-nonhomogeneous)
This study	Expansion and reduction	Finite	General stochastic demand (can be time-nonhomogeneous)

) formulate capacity planning as an MDP and concentrate on expansion under uncertainty, and apart from application cases, few provide rigorous structural characterizations of the optimal policies. Wu and Chuang [15] studied a multigeneration capacity portfolio expansion model involving uncertainties in price, demand, and lifecycle; established a monotone expansion policy; and formulated a revised value iteration algorithm. Wu and Chuang [11] further extended their study to encompass multi-type (flexible versus dedicated) capacity with uncertainties in demand, price, and yield, utilising the structural properties in their previous work [15], and proposed a heuristic search algorithm that improves computational efficiency by approximately 30%. Lin et al. [16] explored expansion and allocation across multiple sites under Markovian demand within a finite-horizon dynamic programming framework embedded into linear programming, although no structural properties were delineated. Mishra et al. [17] investigated a two-period product line expansion/reduction problem characterized by price-dependent stochastic demand, resulting in a news-vendor-like optimal policy in quantile form along with an analytical solution. Serrato et al. [18] modelled the outsourcing of reverse-logistics capacity over a finite horizon with binomial returns and demonstrated a threshold policy through a backward induction algorithm.

Classically, Lindley recursions are associated with inventory and queueing models, yet the same reflected-random-walk dynamics govern capacity headroom under stochastic demand with discrete expand/contract actions. We therefore draw on three inventory studies as methodological antecedents. Recent research by Blancas-Rivera et al. [20] elucidates the conditions under which (s, S) policies are optimal, even in the presence of potentially unbounded costs. Their findings demonstrate that a subsequence of value-iteration minimizers converges to an optimal policy, substantiated through a numerical example. In the realm of periodic-review lost-sales models with positive lead times, van Jaarsveld and Arts [21] proposed the projected inventory-level (PIL) policy. This policy aims to maintain a fixed expected on-hand inventory upon arrival, and it has been analytically proven to dominate constant-order policies. The PIL policy achieves asymptotic optimality as the penalty for lost sales approaches infinity and, in scenarios with exponential demand, as the lead time becomes infinitely long. Their study also reports robust numerical performance. In a related investigation, Yuan et al. [22] establish that within lost-sales systems characterized by stochastic lead times, base-stock policies attain asymptotic optimality when the penalty for lost sales significantly surpasses holding costs, with the incurred costs converging to the system’s optimum.

The studies most pertinent to this research are those conducted by references [17,18]. Serrato et al. [18] investigated outsourcing decisions constrained to expansion scenarios under the assumption of binomially distributed returns. Conversely, Mishra et al. explored a two-period model characterized by demand contingent on price, which resulted in an analytical solution comparable to the newsvendor model. Distinctively, our model extends the model developed by Abduljaleel and AlDurgam [19] to accommodate both expansion and reduction decisions. We incorporate a general stochastic demand model that allows for time-nonhomogeneous demands and derive a monotone structure under realistic cost conditions. Utilising these characteristics, we have developed a revised value iteration algorithm that significantly diminishes computational effort in comparison to the conventional backward value iteration method.

The remainder of this paper is structured as follows. Section 2 provides a formal definition and MDP formulation of the problem at hand. Section 3 presents the structural properties of the model and their advantages with respect to computational efficiency. Based on the structural properties of the MDP model, a modified value iteration algorithm is given in Section 4, and illustrative numerical examples are presented in Section 5. Section 6 concludes the paper.

2. Problem Definition and MDP Model

In this section, we provide the problem definition, notations, and MDP model of the problem at hand.

2.1. Problem Definition

This study addresses a problem of multi-period capacity planning wherein a decision maker is tasked with meeting stochastic dynamic demands. The model assumes there is a Markovian demand process [23]; that is, the probability distribution of the demand in the next time period depends only on the demand realised in the current period. In any time period, the system’s capacity represents the maximum demand it can satisfy internally in that period. Any demand exceeding this capacity incurs a penalty due to emergency processing or expediting. In every time-period, there will be a maintenance cost for maintaining the overall system capacity and an operating cost for the utilised system capacity. At the end of each period, the decision maker decides how much to increase or decrease the system capacity. There is no backordering of demand; that is, at any point in time, if the demand is not met by the system’s capacity, it is processed externally, incurring a penalty cost. The optimal decision policy defines the optimal action in every time period for each possible system state, with the objective of minimising overall system costs over the entire planning horizon.

2.2. Nomenclature

Next, the notations used to construct the MDP model are provided.

$a_t\left({CS}_t,d_t\right)$ denotes the action taken at time $t$ when the system is in state $\left({CS}_t,d_t\right)$. For simplicity, $a_t$ is used interchangeably with $a_t\left({CS}_t,d_t\right)$. Positive $a_t$ values represent increasing the capacity of the system, and negative $a_t$ values represent decreasing the capacity of the system.

${a_t}^{\ast }\left({CS}_t,d_t\right)$ denotes the optimal action taken at time $t$ when the system is in state $\left({CS}_t,d_t\right)$. For simplicity, ${a_t}^{\ast }$ is used interchangeably with ${a_t}^{\ast }\left({CS}_t,d_t\right).$

$A_t\left({CS}_t,d_t\right)$ denotes action space at time $t$ when the system is in state $\left({CS}_t,d_t\right)$. For simplicity, $A_t$ is used interchangeably with $A_t\left({CS}_t,d_t\right).$

$c_1$ denotes the cost of increasing the capacity of the system by one unit; this represents an investment cost ($ per unit of capacity).

$c_2$ denotes the cost of decreasing the capacity of the system by one unit; this represents disinvestment cost/reward ($ per unit of capacity). $c_2$ can take negative values to represent rewards.

$c_3$ denotes capacity maintenance cost ($ per unit of capacity available per time period).

$c_4$ denotes operating cost. This represents the cost incurred in meeting a unit of demand by the existing system capacity ($ per unit of capacity demanded per time period).

$c_5$ denotes shortage cost or the penalty incurred per unit of demand exceeding current system capacity ($ per unit short of capacity per time period).

$c_6$ denotes the cost of salvaging a unit of capacity of the system at the end of the planning horizon, $t=T$, ($ per unit of capacity). To represent rewards, $c_6$ can take negative values.

${CS}_t$ denotes System capacity at time $t$ (units of capacity).

$d_t$ denotes demand for units of capacity at time $t$ (units of capacity).

$MC$ denotes the maximum possible capacity available for the decision maker at any point in time.

$p_{t+1}\left[\left({CS}_{t+1},d_{t+1}\right)|\left({CS}_t,d_t\right),a_t\right]$ denotes the conditional probability the state at time $t+1$ will be $\left({CS}_{t+1},d_{t+1}\right)$, given that the state and action taken at time $t$ were $\left({CS}_t,d_t\right)$ and $a_t$.

$P\left(d_{t+1}|d_t\right)$ denotes the conditional probability the demand at time $t+1$ will be $d_{t+1}$, given that the demand at time $t$ was $d_t$.

$R_t\left[\left({CS}_t,d_t\right),a_t\right]$ denotes the immediate cost incurred when taking an action $a_t$ at time $t$ when the system is in state $\left({CS}_t,d_t\right)$. A positive value of $R_t\left[\left({CS}_t,d_t\right),a_t\right]$ means that $a_t$ resulted in immediate costs, while a negative value represents immediate revenue.

$T$ denotes the length of the planning horizon.

$t$ denotes time period/epoch $t=\mathrm{0,1},2\dots . T$.

$V_t\left(\left({CS}_t,d_t\right),a_t\right)$ denotes the value function when choosing an action $a_t$ at time $t$ when the system is in state $\left({CS}_t,d_t\right)$ subsequently following an optimal decision policy from time $t+1$.

$V_t^{\ast }\left({CS}_t,d_t\right)$ denotes the optimal value function when choosing the optimal decision policy starting from time $t$ when the system is in state $\left({CS}_t,d_t\right).$

2.3. An MDP Model for Multi-Period Capacity Planning

An MDP model consists of system states, a set of actions, transition probabilities between system states, and reward/cost functions that, in their basic form, depend on system state and the action taken [10]. We define these elements for our problem below.

System States: In any time period ($t$) of the planning horizon $\left(T\right)$, the system state is fully described by two state variables—the system capacity and demand experienced during the time period $\left({CS}_t,d_t\right)$. At the beginning of the planning horizon, the system state is assumed to be $\left({CS}_0,d_0\right).$

In instances where component-specific details are irrelevant, such as in expressions involving the value function or action-value function, we denote the state succinctly as $s_t$. However, in scenarios where the argument or proposition is contingent upon the ordering or structure of individual components, such as in the theorems and proofs, we maintain the explicit notation of the components for $s_t$, i.e., $\left({CS}_t,d_t\right)$ and (${CS}_t+a_t,d_{t+1})$ for $s_{t+1}.$

Actions: At the end of every time period $t$, given a system state $\left(s_t\right)$, the decision maker engages in an action $a_t\left(s_t\right)$ (or simply $a_t$) that allows changes in the capacity of the system or keeps the capacity the same by employing the ‘do nothing’ action. The action space $A_t\left(s_t\right)$ (or equally $A_t$ for simplicity) denotes the set of actions available for the decision maker when the system is in state $\left(s_t\right)$ at the end of period $t$, $a_t\left(s_t\right)ϵ A_t\left(s_t\right)$. If the capacity of the system ranges between 0 and $MC$ (the maximum capacity of the system), the values of $a_t$ can be negative integers for a decrease in capacity, zero for the ‘do nothing’ action, or positive integers for an increase in system capacity.

At the end of time $t$ when the system is in state $\left(s_t\right)$, the set of actions available to the decision maker is defined as follows:

$$A_t\left(s_t\right)=\left\{-{CS}_t, -{CS}_t+1,{CS}_t+2\dots MC-{CS}_t\right\}.$$

For instance, if the decision maker chooses the action $-{CS}_t$, the new capacity of the system (${CS}_{t+1}={CS}_t+a_t$) becomes zero.

Transition Probabilities: This MDP element defines the conditional state transition probabilities. Assuming ${CS}_t$ and $d_t$ are independent, the probability of transition to a new system state $\left(s_{t+1}\right)$ in $t+1$, given the current system state$\left(s_t\right)$ and action taken $\left(a_t\right)$, is expressed as

$$p_{t+1}\left[s_{t+1}|s_t,a_t\right]=\left\{\begin{array}{cc}P\left(d_{t+1}|d_t\right)& if {CS}_{t+1}={CS}_t+a_t\\ 0 & otherwise \end{array}\right..$$

(1)

Costs and model dynamics: Each MDP model should have a reward or cost element; the cost elements of the proposed MDP model are described below.

At the end of planning horizon $(t=T)$, the entire system capacity is salvaged. The resulting relations at $T$ are

$$V_T^{\ast }\left(s_T\right)=V_T\left(s_T,a_T\right)=R_T\left[s_T,a_T\right]=c_6{CS}_{T.}$$

(2)

The terminal cost accounts for the cost in salvaging the system capacity at the end of the planning horizon. If there is a reward for salvaging the system, $c_6$ takes a negative value to reflect the reward in the cost function. It is important to clarify that $a_T$ (the action at the end of the planning horizon) is not really a decision variable, as the system simply salvages its entire capacity. We include $a_T$ simply for the sake of model analysis.

At the end of any time period, $t\ne T$, given the state of the system $\left(s_t\right)$ and the action $a_t$carried out by the decision maker, a cost $R_t\left[s_t,a_t\right]$ will be incurred. This is defined as

$$R_t\left[s_t,a_t\right]=c_1{\left(a_t\right)}^{+}+c_2{\left(-a_t\right)}^{+}+{ c}_3\left({CS}_t\right)+c_{4}min\left(d_t, {CS}_t\right)+{ c}_5{\left(d_t-{CS}_t\right)}^{+}.$$

(3)

In Equation (3), the first term, $c_1{\left(a_t\right)}^{+}$, accounts for the costs incurred for increasing the capacity of the system (investment cost), that is, whenever $a_t$is positive. The second term, $c_2{\left(-a_t\right)}^{+}$, represents costs for any decrease in capacity (disinvestment), that is, whenever $a_t$ is negative. The third term, ${ c}_3\left({CS}_t\right)$, represents the costs for maintaining the system capacity; the fourth term, $c_{4}min\left(d_t, {CS}_t\right)$, represents operating cost in period $t$; and the fifth term,${ c}_5{\left(d_t- {CS}_t\right)}^{+}$, accounts for the penalty incurred when the demand exceeds system capacity in period $t.$

The decision maker establishes the decision policy, which involves selecting an action for each time period within the planning horizon, depending on the current state of the system. The policy is designed to minimize the total expected cost throughout the entire planning horizon. The optimal value function that gives the minimum expected cost, from period $t$ to the end of the planning horizon when following an optimal decision policy, is given as follows:

$$V_t^{\ast }\left(s_t\right)=\begin{array}{c}min\\ a_t\in A_t\left(\right)\end{array}\left\{\begin{array}{c}min\\ a_t\in A_t\left(\right)\end{array}\left\{R_t\left[s_t,a_t\right]+\sum _{d_{t+1}}{p}_{t+1}\left[s_{t+1}|s_t,a_t\right]V_{t+1}^{\ast }\left(s_{t+1}\right)\right\}\right\}.$$

(4)

Using Equation (1), this can be also rewritten as

$$V_t^{\ast }\left(s_t\right)=\begin{array}{c}min\\ a_t\in A_t\left(\right)\end{array}\left\{R_t\left[s_t,a_t\right]+\sum _{d_{t+1}}P\left(d_{t+1}|d_t\right)V_{t+1}^{\ast }\left({CS}_t+a_t,d_{t+1}\right)\right\}.$$

(5)

Hence, the optimal action ${a_t}^{\ast }\left(s_t\right)$ is expressed as follows:

$${a_t}^{\ast }\left(s_t\right)={\displaystyle \genfrac{}{}{0pt}{}{arg\min a_t}{a_t\in A_t\left(s_t\right)}}\left\{R_t\left[s_t,a_t\right]+\sum _{d_{t+1}}P\left(d_{t+1}|d_t\right)V_{t+1}^{\ast }\left({CS}_t+a_t,d_{t+1}\right)\right\}.$$

(6)

The value function that yields the expected cost from period $t$ to the end of planning horizon $T$—when carrying out action $a_t$ at the end of period $t$ and henceforth following the optimal decision policy—is expressed as $V_t\left(s_t,a_t\right)$:

$$V_t\left(s_t,a_t\right)=R_t\left[s_t,a_t\right]+{\sum }_{d_{t+1}}P\left(d_{t+1}|d_t\right){ V}_{t+1}^{\ast }\left({CS}_t+a_t,d_{t+1}\right).$$

(7)

It should be noted that Equations (4)–(7) are derived directly based on the standard Bellman optimality equation.

2.4. The Value Iteration Algorithm

The optimal decision policy for a finite-horizon MDP model can be determined using the backward value iteration algorithm [10]. Hence, for our model, if the demand takes integer values of $[0,x]$, then the number of iterations required is

$$T\times {\left(x+1\right)}^3.$$

For each epoch, there are ${\left(x+1\right)}^2$ possible states, and for each state, there are $\left(x+1\right)$ possible actions. In Section 4, we present some structural properties of our model and discuss the advantages of the model’s structural properties in terms of computational effort.

3. A Structured Optimal Capacity-Planning Policy

The existence of a structured optimal policy has two main advantages: it reduces the computational effort required to determine the optimal policy, and it simplifies the application of the optimal policy. In terms of the cost elements, all the structural properties derived in this paper and proven in the Appendix A are based only on the following two realistic conditions:

$$c_1\ge max\left({0,-c}_2\right),$$

$$c_5>c_4+c_3.$$

The first condition means that if revenue is generated in the context of a decreasing system capacity, it will be less than the cost of increasing system capacity. The second condition means that the shortage cost per unit of capacity is greater than the sum of the maintenance and operating costs per unit of capacity. Below, the different structural properties of the optimal policy are provided, considering the stated cost assumptions.

Theorem 1:

In any time period $t$, given an arbitrary state $\left({CS}_t=CS1,d_t\right)$, if the optimal action is increasing the system’s capacity to level $g (g>CS1)$, then for all states $\left({CS}_t=CS2,d_t\right)$, where $CS1 \le CS2 \le g$, increasing system capacity to reach the same capacity level $g$ would be an optimal action. Mathematically, if ${a_t}^{\ast }\left({CS}_t=CS1,d_t\right)=g-CS1$ and $g\ge CS1$, then ${a_t}^{\ast }\left({CS}_t=CS2,d_t\right)=g-CS2 \forall CS1\le CS2\le g.$

Practical meaning:

Once a capacity target is justified, being further below that target cannot be optimal; that is, partial expansions should be avoided.

Proof: 

The proof of Theorem 1 is given in Appendix A at the end of the paper. □

Theorem 2:

In any time period $t$, given an arbitrary system state $\left({CS}_t=CS2,d_t\right)$, if the optimal action is decreasing system capacity to level $e\ge 0$, then for all states $\left({CS}_t=CS1,d_t\right)$, where $e\le CS1\le CS2$, an optimal action would be to decrease system capacity to reach the same capacity level $e$. Mathematically, if ${a_t}^{\ast }\left({CS}_t=CS2,d_t\right)=e-CS2$ and $e\le CS2$, then, ${a_t}^{\ast }\left({CS}_t=CS1,d_t\right)=e-CS1 \forall e\le CS1 \le CS2.$

Practical meaning:

Once a contraction target is justified, being further above that target cannot be optimal; that is, partial contractions should be avoided.

Proof: 

The proof of Theorem 2 is similar to that for Theorem 1, and it is given in Appendix A at the end of the paper. □

The following Lemma 1 is used to arrive at our Proposition 1, which is stated below.

Lemma 1:

If $g$ is a superadditive (subadditive) function for X $\times$ Y and for each $x \in$ X, $\underset{y\in Y}{\mathit{min}}g\left(x,y\right)$ exists, then

$$f\left(x\right)=min\left\{y^{\prime }\in \underset{y\in Y}{\mathit{arg\; min}}g\left(x,y\right)\right\}$$

 is monotonously non-increasing (non-decreasing) in $x.$

Proof. 

The proof of this Lemma is given by reference [10]. □

Lemma 2 is used in the proof for Proposition 1.

Lemma 2:

In any time period $t$, the function

$$c_{4}min\left(d_t, {CS}_{t-1}+a_{t-1}\right)+c_5{\left(d_t-{CS}_{t-1}-a_{t-1}\right)}^{+}$$

 is superadditive in ${CS}_{t-1}\times a_{t-1}$∀$d_t.$

Proof: 

The proof of this Lemma is straightforward, and it is given in the Appendix A. □

Theorem 3, which will be stated later, is based on Lemma 1 and Proposition 1.

Proposition 1:

In the given problem definition, for any time period t, $V_t\left(\left({CS}_t,d_t\right),a_t\right)$is superadditive in ${CS}_t \times a_t$ ∀ $d_t$. That is, for any capacity levels $CS2$ and $CS1$, such that $CS2\ge CS1$, and for any actions ${a2}_t$ and ${a1}_t$ in $A_t$, such that ${a2}_t\ge {a1}_t$,

$$V_t\left(\left(CS2,d_t\right),{a2}_t\right)-V_t\left(\left(CS2,d_t\right),{a1}_t\right)\ge V_t\left(\left(CS1,d_t\right),{a2}_t\right)-V_t\left(\left(CS1,d_t\right),{a1}_t\right).$$

Proof: 

The proof of this proposition is given in Appendix A. □

The main result of this paper, Theorem 3, is stated next.

Theorem 3:

For a given $d_t$, ${a_t}^{\ast }\left({CS}_t,d_t\right)$ is nonincreasing in ${CS}_t$.

Practical meaning:

For a fixed period and demand state, the optimal adjustment is nonincreasing in current capacity (with a greater installed capacity, one should never expand more).

Proof: 

By rewriting the equations for $V_t^{\ast }\left({CS}_t,d_t\right)$, $a_t\left({CS}_t,d_t\right)$, and $V_t\left(\left({CS}_t,d_t\right),a_t\right)$ given in the previous section, we obtain

$$V_t^{\ast }\left({CS}_t,d_t\right)=\begin{array}{c}min\\ a_t\in A_t\left(\right)\end{array}\left\{R_t\left[\left({CS}_t,d_t\right),a_t\right]+\sum _{d_{t+1}}P\left(d_{t+1}|d_t\right)V_{t+1}^{\ast }\left({CS}_t+a_t,d_{t+1}\right)\right\},$$

$${a_t}^{\ast }\left({CS}_t,d_t\right)={\displaystyle \genfrac{}{}{0pt}{}{arg\min a}{a_t\in A_t\left(\right)}}\left\{R_t\left[\left({CS}_t,d_t\right),a_t\right]+\sum _{d_{t+1}}P\left(d_{t+1}|d_t\right)V_{t+1}^{\ast }\left({CS}_t+a_t,d_{t+1}\right)\right\},$$

$$V_t\left(\left({CS}_t,d_t\right),a_t\right)=R_t\left[\left({CS}_t,d_t\right),a_t\right]+\sum _{d_{t+1}}P\left(d_{t+1}|d_t\right){ V}_{t+1}^{\ast }\left({CS}_t+a_t,d_{t+1}\right).$$

We use Lemma 1 to acquire the result of Theorem 3. As per Lemma 1, if $V_t\left(\left({CS}_t,d_t\right),a_t\right)$ is superadditive in ${CS}_t \times a_t$∀$d_t$, then Theorem 3 follows. Proposition 1 proves that $V_t\left(\left({CS}_t,d_t\right),a_t\right)$ is superadditive in ${CS}_t \times a_t$∀$d_t$. Through Theorem 3, the computational effort is greatly reduced.

If the demand takes values from $\left[0,x\right]$, the number of iterations required using Proposition 1 is less.

Owing to Theorems 1–3, the number of iterations required to find an optimal decision policy reduces. If the demand takes values from $\left[0,x\right]$, the maximum number of iterations will be

$$T\left(3x+1\right)\left(x+1\right).$$

After searching in $\left(x+1\right)$ actions at both the first and last capacity levels $\left(x+1\right)\times \left(2x+2\right)$, Theorems 1–3 are applied to determine the optimal action at the remaining capacity levels $\left(x+1\right)\times \left(x-1\right)$. One can argue that what we have presented is more of a loose upper bound.

We can delineate additional structural properties of the defined problem. Theorem 4 will help in the search for the optimal action in the initial states. □

Theorem 4:

In the defined problem, $V_t\left(\left({CS}_t,d_t\right),a_t\right)$ is convex in $a_t\forall {CS}_t,d_t,t$.

Practical meaning:

For any period, demand state, and current capacity, the look-ahead cost as a function of the adjustment is convex.

Proof: 

The proof of this theorem is given in the Appendix A. □

Theorem 4 further reduces the number of iterations required in order to determine the optimal decision policy. This issue is addressed in Section 5.

We put forward Lemma 3 to prove Proposition 2, which is used for the purpose of proving Theorem 5.

Lemma 3:

The function

$$c_{4}min\left(d_t, {CS}_{t-1}+a_{t-1}\right)+c_5{\left(d_t-{CS}_{t-1}-a_{t-1}\right)}^{+}.$$

 is subadditive in $d_t \times a_{t-1}$ ∀ ${CS}_{t-1}.$

Proof: 

The proof is given in Appendix A. □

Proposition 2:

In the presented MDP model, if the demand transition matrix exhibits first-order stochastic dominance (that is, if the probability that the demand exceeds a specific value in the next state does not decrease with an increase in the demand experienced in the current state), then $V_t\left(\left({CS}_t,d_t\right),a_t\right)$ is subadditive in $d_t \times a_t$ ∀ ${CS}_t$. That is, for any two demands ${\mathrm{d}2}_{\mathrm{t}}, {\mathrm{d}1}_{\mathrm{t}}$ such that ${\mathrm{d}2}_{\mathrm{t}}\ge {\mathrm{d}1}_{\mathrm{t}}$, and actions ${\mathrm{a}2}_{\mathrm{t}}, {\mathrm{a}1}_{\mathrm{t}}$ in ${\mathrm{a}}_{\mathrm{t}}$, such that ${\mathrm{a}2}_{\mathrm{t}}\ge {\mathrm{a}1}_{\mathrm{t}}$,

$$V_t\left(\left({CS}_t,{d2}_t\right),{a2}_t\right)-V_t\left(\left({CS}_t,{d2}_t\right),{a1}_t\right)\le V_t\left(\left({CS}_t,{d1}_t\right),{a2}_t\right)-V_t\left(\left({CS}_t,{d1}_t\right),{a1}_t\right).$$

Proof: 

The proof is given in Appendix A. □

Theorem 5:

If the demand transition follows first-order stochastic dominance, then the optimal action does not decrease with an increase in $d_t$ in the state definition. In other words, for a given ${CS}_t$, $a_t\left({CS}_t,d_t\right)$ is nondecreasing in $d_t$ if the demand transition follows first-order stochastic dominance. That is, $\forall d1, d2$ in $d_t$ such that $d2\le d1$, if $P\left(d_{t+1}>d^{\prime }|d2\right)\ge P\left(d_{t+1}>d^{\prime }|d1\right) \forall d^{\prime },t$. Then, $a_t\left({CS}_t,d2\right)\ge a_t\left({CS}_t,d1\right) \forall {CS}_t,t.$

Practical meaning:

Under FOSD, higher-demand states never call for smaller moves; that is, expansion should not be reduced and further cuts should not be made when demand is in a state with a higher demand level.

Proof: 

Because of Lemma 1, Theorem 5 follows if ${\mathrm{V}}_{\mathrm{t}}\left(\left({\mathrm{C}\mathrm{S}}_{\mathrm{t}},{\mathrm{d}}_{\mathrm{t}}\right),{\mathrm{a}}_{\mathrm{t}}\right)$ is subadditive in ${\mathrm{d}}_{\mathrm{t}}\times {\mathrm{a}}_{\mathrm{t}}$ ∀${\mathrm{C}\mathrm{S}}_{\mathrm{t}}$. Proposition 2 proves that $V_t\left(\left({CS}_t,d_t\right),a_t\right)$ is subadditive in $d_t \times a_t$ ∀${CS}_t$. □

4. A Revised Value Iteration Algorithm

This section presents a modified value iteration algorithm that will further reduce the computational efforts compared with the standard backward value iteration algorithm [10]. The number of iterations required to obtain the optimal policy is, at most,

$$T\left(2x\right)\left(x+1\right).$$

For a problem definition where the demand takes values from $\left[0,x\right]$, a search is performed in $\left(x+1\right)$ actions together for both the first and last capacity levels $\left(x+1\right)\times \left(x+1\right)$; then, Theorems 1, 2, 3, and 4 are applied to determine the optimal action at the remaining capacity levels $\left(x+1\right)\times \left(x-1\right)$. Again, this is more of a loose upper bound. In case of large values of $x$, we can employ the modified Golden section search method or modified Fibonacci search, the latter of which is used for discrete functions to reduce the steps/time required to determine the optimal policy.

In Section 4.1, based on Theorems 1–4, we present a modified value iteration algorithm where we do not impose any condition on the demand transition matrix; hence, our model and its structural properties apply to the case of time-nonhomogeneous transitions. In addition, if the state transition matrix follows first-order stochastic dominance, we will obtain additional structural properties (Proposition 2 and Theorem 5); then, the algorithm in Section 4.2 applies.

4.1. Modified Value Iteration Algorithm 1

Next, we present a modified value iteration algorithm for the problem at hand. Additionally, we highlight the links between the proposed algorithm and the theorems presented in Section 4.

Algorithm 1: Pseudocode

Input: Horizon T; Maximum system capacity $MC$; maximum demand max_demand; Demand transition matrix $P\left(d_{t+1}|d_t\right)$; Cost components: $c_1, c_2, c_3, c_4, c_5, c_6$; Output: Optimal action table ${a_t}^{\ast }\left({CS}_t, d_t\right)$ and optimal value table $V_t^{\ast }\left({CS}_t,d_t\right)$ Initialize: Set $t=T$, compute $V_T^{\ast }\left({CS}_T,d_T\right)= c_6{CS}_{T.}$ For $t=T-1$ down to 1 do:       For: $d_t=0$ to max_demand do:       Find optimal action ${a_t}^{\ast }\left({CS}_t,d_t\right)$ and value $V_t^{\ast }\left({CS}_t,d_t\right)$ for ${CS}_t=0$ (convexity)           Set $x_{opt}=0$ and $V_{current}=V_t\left(\left(0,d_t\right),0\right)$           For x = 1 to $MC$ do:               If $V_t\left(\left(0,d_t\right),x\right)\le V_{current}$ then:                    Set $x_{opt}=x$ and $V_{current}=V_t\left(\left(0,d_t\right),x\right)$               Else: break           Set ${a_t}^{\ast }\left(0,d_t\right)= x_{opt}$ and $V_t^{\ast }\left(0, d_t\right)= V_{current}$       For ${CS}_t=1$ to ${a_t}^{\ast }\left(0,d_t\right)$ do: (Theorem 1)           Set ${a_t}^{\ast }\left({CS}_t,d_t\right)= {a_t}^{\ast }\left(0,d_t\right)-{ CS}_t$           Set $V_t^{\ast }\left({CS}_t, d_t\right)= V_t\left(\left({CS}_t,d_t\right),{a_t}^{\ast }\left({CS}_t,d_t\right)\right)$       Find optimal action ${a_t}^{\ast }\left({CS}_t,d_t\right)$ and value $V_t^{\ast }\left({CS}_t,d_t\right)$ for ${CS}_t=MC$ (convexity)           Set $y_{start}={a_t}^{\ast }\left(0,d_t\right)-MC$           Set $y_{opt}=y_{start}$ and $V_{current}=V_t\left(\left(MC,d_t\right),y_{start}\right)$           For $y= y_{start}+1$ to 0 do:               If $V_t\left(\left(MC,d_t\right),y\right)\le V_{current}$ then:                    Set $y_{opt}=y$ and $V_{current}=V_t\left(\left(MC,d_t\right),y\right)$               Else: break           Set ${a_t}^{\ast }\left(MC,d_t\right)= y_{opt}$ and $V_t^{\ast }\left(MC, d_t\right)= V_{current}$       Set $L=MC+ {a_t}^{\ast }\left(MC,d_t\right)$       For ${CS}_t=L$ to $MC-1$ do: (Theorem 2)           Set ${a_t}^{\ast }\left({CS}_t,d_t\right)= L-{ CS}_t$           Set $V_t^{\ast }\left({CS}_t, d_t\right)= V_t\left(\left({CS}_t,d_t\right),{a_t}^{\ast }\left({CS}_t,d_t\right)\right)$       For ${CS}_t={a_t}^{\ast }\left(0,d_t\right)+1$ to $L-1$ do: (Theorem 3)           Set ${a_t}^{\ast }\left({CS}_t,d_t\right)= 0$           Set $V_t^{\ast }\left({CS}_t, d_t\right)= V_t\left(\left({CS}_t,d_t\right),0\right)$

4.2. Modified Value Iteration Algorithm 2

In case the demand transition matrix $P\left(d_{t+1}|d1\right)$ corresponds to first-order stochastic dominance, we can further improve our modified value iteration algorithm, as follows.

Algorithm 2: Pseudocode

Input: Horizon T; Maximum system capacity $MC$; maximum demand max_demand; Demand transition matrix $P\left(d_{t+1}|d_t\right)$; Cost components: $c_1, c_2, c_3, c_4, c_5, c_6$; Output: Optimal action table ${a_t}^{\ast }\left({CS}_t, d_t\right)$ and optimal value table $V_t^{\ast }\left({CS}_t,d_t\right)$ Initialize: Set $t=T$, compute $V_T^{\ast }\left({CS}_T,d_T\right)= c_6{CS}_{T.}$ For $t=T-1$ down to 1 do:       For: $d_t=0$ to max_demand do:       Find optimal action ${a_t}^{\ast }\left(0,d_t\right)$ and value $V_t^{\ast }\left(0,d_t\right)$ (convexity & Theorem 5)           If $d_t=0$ then:               Set $x_{start}=0$           Else: Set $x_{start}={a_t}^{\ast }\left(0,d_t\right)$           Set $x_{opt}=x_{start}$ and $V_{current}=V_t\left(\left(0,d_t\right),x_{start}\right)$           For x = $x_{start}+1$ to $MC$ do:               If $V_t\left(\left(0,d_t\right),x\right)\le V_{current}$ then:                    Set $x_{opt}=x$ and $V_{current}=V_t\left(\left(0,d_t\right),x\right)$               Else: break           Set ${a_t}^{\ast }\left(0,d_t\right)= x_{opt}$ and $V_t^{\ast }\left(0, d_t\right)= V_{current}$       For ${CS}_t=1$ to ${a_t}^{\ast }\left(0,d_t\right)$ do: (Theorem 1)           Set ${a_t}^{\ast }\left({CS}_t,d_t\right)= {a_t}^{\ast }\left(0,d_t\right)-{ CS}_t$           Set $V_t^{\ast }\left({CS}_t, d_t\right)= V_t\left(\left({CS}_t,d_t\right),{a_t}^{\ast }\left({CS}_t,d_t\right)\right)$       Find optimal action ${a_t}^{\ast }\left({CS}_t,d_t\right)$ and value $V_t^{\ast }\left({CS}_t,d_t\right)$ for ${CS}_t=MC$ (convexity)           If $d_t=0$ then:               Set $y_{start}={a_t}^{\ast }\left(0,d_t\right)-MC$           Else:               Set $y_{start}=max\left({a_t}^{\ast }\left(0,d_t\right)-MC, {a_t}^{\ast }\left(MC,d_t-1\right)\right)$ Set $y_{opt}=y_{start}$ and $V_{current}=V_t\left(\left(MC,d_t\right),y_{start}\right)$           For $y= y_{start}+1$ to 0 do:               If $V_t\left(\left(MC,d_t\right),y\right)\le V_{current}$ then:                    Set $y_{opt}=y$ and $V_{current}=V_t\left(\left(MC,d_t\right),y\right)$               Else: break           Set ${a_t}^{\ast }\left(MC,d_t\right)= y_{opt}$ and $V_t^{\ast }\left(MC, d_t\right)= V_{current}$       Set $L=MC+ {a_t}^{\ast }\left(MC,d_t\right)$       For ${CS}_t=L$ to $MC-1$ do: (Theorem 2)           Set ${a_t}^{\ast }\left({CS}_t,d_t\right)= L-{ CS}_t$           Set $V_t^{\ast }\left({CS}_t, d_t\right)= V_t\left(\left({CS}_t,d_t\right),{a_t}^{\ast }\left({CS}_t,d_t\right)\right)$       For ${CS}_t={a_t}^{\ast }\left(0,d_t\right)+1$ to $L-1$ do: (Theorem 3)           Set ${a_t}^{\ast }\left({CS}_t,d_t\right)= 0$           Set $V_t^{\ast }\left({CS}_t, d_t\right)= V_t\left(\left({CS}_t,d_t\right),0\right)$

4.3. Values of Structural Properties

To quantify the benefits of the proposed structural properties and the resulting modified value iteration algorithms, we conducted 100 numerical experiments. Each experiment modelled a system with a capacity range of 0 to 10 units over a planning horizon of 10 discrete periods. Demand distributions were generated to satisfy first-order stochastic dominance, and all other problem parameters were randomly sampled to ensure generality. The experimental setup is summarized as follows:

• Using R, all draws use runif() and are then rounded to the nearest integer (banker’s rounding):

  -

  c1: Uniform on $\left[\mathrm{10,100}\right]\to$ integer in $\{10,\dots ,100\}$.

  -

  $\mathrm{c}2$: Uniform on $[0,\mathrm{c}1-1]\to$ integer in $\{0,\dots ,c1-1\}$ (so $c2<c1$).

  -

  c3: Uniform on $\left[\mathrm{10,100}\right]\to$ integer in $\{10,\dots ,100\}$.

  -

  c4: Uniform on $\left[\mathrm{10,100}\right]\to$ integer in $\{10,\dots ,100\}$.

  -

  c5: Uniform on $[c3+c\mathrm{4,250}]\to$ integer in $\{c3+c4,\dots ,250\}$ (always $\ge$ maintenance + labour).

  -

  c6: (salvage “cost”): -round (runif $(...,\mathrm{0,6})$) $\to$ integer in $\{0,-1,-2,-3,-4,-5,-6\}$ (i.e., a non-positive per-unit term).

• Transition matrix $P_d$ (size $11\times 11$, since CSM = 10)
• Row 1: Draw $n$ i.i.d. uniforms, normalize them to sum to 1, round each one to 2 decimals, and then set the last entry to the residual $1-{\sum }_{j=1}^{n-1} p_j$.
  Practically, the first $n-1$ entries lie on a 0.01 grid in $\left[\mathrm{0,1}\right]$; the last entry is whatever makes the row sum exactly 1.
• Rows 2…n: This is constructed to keep each row’s CDF no larger than the previous row’s CDF (first-order stochastic dominance in that direction). For each column $j<n$,

$$p_j\sim \mathrm{Uniform} \left(0, \mathrm{prev}\_\mathrm{cdf} [j]-\sum _{k<j} p_k\right),$$

which is then truncated to 2 decimals via floor $(\dots \ast 100)/100$. The last entry is again the residual $1-\sum _{j=1}^{n-1} p_j$, so each $p_j$ lies on a 0.01 grid between 0 and its row-specific cap set by the previous row’s CDF.

• Number of experiments: Num Experiments $=100$ (independent re-draws of the above per experiment).
• Capacity/demand state space: $\mathrm{C}\mathrm{S}\mathrm{M}=10\Rightarrow$ states $0,\dots ,10$.
• Horizon: $\mathrm{H}=10$.

In a brute-force approach, the total number of computations required is substantial: (11 capacity states) $\times$ (11 demand states) $\times$ (11 actions at each state) $\times$ (9 time periods) = 11,979 calculations

In contrast to the brute-force approach, Figure 1 illustrates that the proposed modified value iteration algorithms, which leverage the structural properties of the MDP, substantially reduce the computational effort required to compute the optimal decision policies.

• Algorithm 1 operates without relying on any structural assumptions regarding the demand transition matrix. By leveraging the structural properties established in Theorem 1 (structuredness of capacity expansion decisions), Theorem 2 (structuredness of capacity reduction decisions), and Theorem 3 (monotonicity of optimal actions with respect to the capacity state), it achieves an average reduction of 82% in the required number of computational steps.
• Algorithm 2 extends these results by leveraging additional structural properties. Beyond the preceding theorems, it incorporates Theorem 4, which establishes the monotonicity of optimal actions with respect to the demand state under first-order stochastic dominance of the demand transition probabilities. Exploiting this structure yields substantial gains in computational efficiency, with average reductions in computational effort exceeding 87%.

Analogous to threshold maintenance policies and base-stock policies in maintenance and production, structured policies make MDP-based capacity planning intuitive for decision makers. They turn state-by-state optimization into clear rules that managers can check and follow. In the context of the monotonicity results (Theorems 1 and 2), the system admits two targets: a level to expand to and a level to shrink to. As a result, one only needs to solve exactly at the capacity bounds; for any interior capacity, the optimal move is simply the gap to the relevant target. This collapses the search space, speeds up computation substantially (with about an 82% average reduction when employing Algorithm 1), and yields straightforward, defensible guidance on when and how much to adjust capacity.

5. Numerical Examples

This section presents two examples to demonstrate the structural properties of the model presented in this paper.

Example 1. 

Consider a problem where a system adjusts its capacity to fulfil demands. Any unit increase in the capacity costs the system USD 13, while any reduction in the capacity costs the system USD 4. The cost of maintaining the unit system capacity per period is USD 12. The cost of using one unit for the capacity to meet demand (operating cost) during a period is USD 24. The penalty incurred for each unit of demand exceeding the system capacity in a period is USD 65. The cost of salvaging the unit capacity at the end of planning horizon is −USD 3 (represents revenue). We need to find an optimal decision policy for capacity planning for a planning horizon of 10 periods. The maximum capacity level of the system is 10 units, and capacity decisions are made at the end of every period.

Problem parameters

$$c_1:13\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{c}_2:4\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{c}_3:12\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{c}_4:25$$

$$c_5:65\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{c}_6:-3\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}T:10\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}MC:10$$

The demand transition matrix $P\left(d_{t+1}|d_t\right)$is

$$\begin{array}{cccccccccccc}{d}_t|d_{t+1}& 0& 1& 2& 3& 4& 5& 6& 7& 8& 9& 10\\ 0& 0.2& 0.1& 0.3& 0& 0.1& 0& 0& 0& 0& 0& 0.3\\ 1& 0.1& 0.2& 0.2& 0.4& 0.1& 0& 0& 0& 0& 0& 0\\ 2& 0.1& 0.1& 0.2& 0& 0.1& 0.1& 0& 0& 0.4& 0& 0\\ 3& 0.1& 0.1& 0.2& 0& 0.2& 0.1& 0& 0& 0& 0& 0.3\\ 4& 0.1& 0.1& 0& 0& 0.2& 0.2& 0& 0& 0& 0.3& 0.1\\ 5& 0.1& 0& 0& 0.2& 0.2& 0.2& 0.1& 0& 0& 0.2& 0\\ 6& 0& 0.1& 0.1& 0.2& 0.1& 0.2& 0.1& 0.1& 0.1& 0& 0\\ 7& 0& 0& 0.1& 0& 0.1& 0.1& 0.1& 0.1& 0.3& 0.2& 0\\ 8& 0& 0& 0& 0.1& 0& 0.1& 0.1& 0.1& 0.2& 0.2& 0.2\\ 9& 0& 0& 0& 0& 0& 0.1& 0.1& 0.2& 0.2& 0.2& 0.2\\ 10& 0& 0& 0& 0& 0.1& 0.1& 0.1& 0.1& 0.2& 0.2& 0.2\end{array}$$

The objective is to minimize the total cost over the planning horizon.

Optimal solution

The optimal solution to Example 1 is provided and demonstrated in

Table 2. Example 1, optimal actions ${a_t}^{\ast }\left({CS}_t,d_t\right)$ at t = 1, 2, 3.

${\mathrm{d}}_{\mathrm{t}}$\${\mathrm{C}\mathrm{S}}_{\mathrm{t}}$	0	1	2	3	4	5	6	7	8	9	10
0	4	3	2	1	0	0	0	0	0	0	0
1	3	2	1	0	0	0	0	0	0	−1	−2
2	8	7	6	5	4	3	2	1	0	−1	−2
3	5	4	3	2	1	0	0	0	0	0	0
4	8	7	6	5	4	3	2	1	0	0	−1
5	5	4	3	2	1	0	0	0	0	0	−1
6	5	4	3	2	1	0	0	0	0	−1	−2
7	8	7	6	5	4	3	2	1	0	0	−1
8	8	7	6	5	4	3	2	1	0	0	−1
9	8	7	6	5	4	3	2	1	0	0	−1
10	8	7	6	5	4	3	2	1	0	0	−1

,

Table 3. Example 1, optimal actions at t = 4 (${a_4}^{\ast }\left({CS}_4,d_4\right)$).

${\mathrm{d}}_4$\${\mathrm{C}\mathrm{S}}_4$	0	1	2	3	4	5	6	7	8	9	10
0	4	3	2	1	0	0	0	0	0	0	0
1	3	2	1	0	0	0	0	−1	−2	−3	−4
2	8	7	6	5	4	3	2	1	0	−1	−2
3	5	4	3	2	1	0	0	0	0	0	0
4	8	7	6	5	4	3	2	1	0	0	−1
5	5	4	3	2	1	0	0	0	0	0	−1
6	5	4	3	2	1	0	0	0	0	−1	−2
7	8	7	6	5	4	3	2	1	0	0	−1
8	8	7	6	5	4	3	2	1	0	0	−1
9	8	7	6	5	4	3	2	1	0	0	−1
10	8	7	6	5	4	3	2	1	0	0	−1

and

Table 4. Example 1, optimal actions at t = 5 (${a_5}^{\ast }\left({CS}_5,d_5\right)$).

${\mathrm{d}}_5$\${\mathrm{C}\mathrm{S}}_5$	0	1	2	3	4	5	6	7	8	9	10
0	4	3	2	1	0	0	0	0	0	0	0
1	3	2	1	0	0	0	−1	−2	−3	−4	−5
2	8	7	6	5	4	3	2	1	0	−1	−2
3	5	4	3	2	1	0	0	0	0	0	0
4	6	5	4	3	2	1	0	0	0	0	−1
5	5	4	3	2	1	0	0	0	0	0	−1
6	5	4	3	2	1	0	0	0	0	−1	−2
7	8	7	6	5	4	3	2	1	0	0	−1
8	8	7	6	5	4	3	2	1	0	0	−1
9	8	7	6	5	4	3	2	1	0	0	−1
10	8	7	6	5	4	3	2	1	0	0	−1

and Figure 1 and Figure 2. The tables demonstrate that the optimal actions are monotonic for all levels of system capacity and do not change depending on the demand, as shown in Figure 2.

The same pattern repeats for the optimal decision rules for periods 6, 7, and 8, which we will skip. Figure 2 and Figure 3 demonstrate the optimal decision rules when $t=1.$.

Upon analysing the optimal decision at different time epochs, we find that the result is in accordance with our listed theorems:

• After determining ${a_t}^{\ast }\left({CS}_t=0,d_t\right)$, we see that, with every unit increase in ${CS}_t$, ${a_t}^{\ast }\left({CS}_t,d_t\right)$ decreases by 1 unit until it reaches 0. This follows from Theorem 1.
• After determining ${a_t}^{\ast }\left({CS}_t=MC,d_t\right)$, we see that, with every unit decrease in ${CS}_t$, ${a_t}^{\ast }\left({CS}_t,d_t\right)$ increases by 1 unit until it reaches 0. This follows from Theorem 2.
• For a given $d_t$, ${a_t}^{\ast }\left({CS}_t,d_t\right)$ does not increase with ${CS}_t$. This follows from Theorem 3.

Theorems 1–3 allow a simpler representation of the policy in Figure 2, which is given in Figure 4. Essentially, knowing the optimum policy at maximum and minimum capacity levels for a demand defines the optimal policy for the remaining capacity levels.

Example 2.

In this example, we work on the same problem parameters as in Example 1, but, now, we replace the demand transition matrix with one that has first-order stochastic dominance, as given below.

$$\begin{array}{cccccccccccc}{d}_t|d_{t+1}& 0& 1& 2& 3& 4& 5& 6& 7& 8& 9& 10\\ 0& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0\\ 1& 0& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1\\ 2& 0& 0& 0.2& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1& 0.1\\ 3& 0& 0& 0.1& 0.2& 0.1& 0.1& 00.1& 0.1& 0.1& 0.1& 0.1\\ 4& 0& 0& 0& 0.2& 0.1& 0.2& 0.1& 0.1& 0.1& 0.1& 0.1\\ 5& 0& 0& 0& 0& 0.1& 0.2& 0.2& 0.2& 0.1& 0.1& 0.1\\ 6& 0& 0& 0& 0& 0& 0.2& 0.2& 0.3& 0.1& 0.1& 0.1\\ 7& 0& 0& 0& 0& 0& 0.1& 0.3& 0.3& 0.1& 0.1& 0.1\\ 8& 0& 0& 0& 0& 0& 0.1& 0.2& 0.3& 0.1& 0.2& 0.1\\ 9& 0& 0& 0& 0& 0& 0.1& 0.1& 0.3& 0.2& 0.2& 0.1\\ 10& 0& 0& 0& 0& 0& 0.1& 0.1& 0.2& 0.2& 0.2& 0.2\end{array}$$

The resulting optimal solution at $t=1$ is demonstrated in Figure 4 and Figure 5.

Upon analysing the optimal decision at different time epochs, we find that the result is in accordance with our listed theorems:

• Theorems 1–3 are upheld, as described in Example 1; and since this problem definition has a demand matrix satisfying first-order stochastic dominance, for a given ${CS}_t$, ${a_t}^{\ast }\left({CS}_t,d_t\right)$ does not decrease with $d_t$. This follows from Theorem 4. Figure 3 and Figure 5 demonstrate that the results become more structured when the conditions of Theorem 4 hold true.

6. Summary

This study presents the formulation of a finite-horizon Markov decision process (MDP) for addressing capacity planning challenges characterized by time-variant Markovian demand. The approach integrates both capacity expansion and contraction within a generic cost structure. This research identifies two fundamental and practical cost conditions that facilitate the development of a simplified policy structure. Furthermore, when demand transitions adhere to first-order stochastic dominance, an additional property of monotonicity is established. We also leverage the property of convexity to enhance the value iteration process, thereby reducing computational effort.

In finite-horizon, time-varying contexts, optimal policies may appear counterintuitive. This study has established sufficient cost conditions (including when assuming there is first-order stochastic dominance (FOSD)), where these policies simplify to straightforward, verifiable rules. (i) State-contingent targets indicate that if expanding to level g is optimal at a certain capacity, the same target is applicable from any lower capacity; conversely, if contracting to level e is optimal, the same target applies for any higher capacity. (ii) There is a monotonicity with respect to capacity such that with increased installed capacity, the prescribed expansions are not larger. (iii) There is monotonicity across demand states, where under FOSD, higher demand states do not necessitate smaller adjustments. This policy is computationally efficient, facilitating routine and defensible decisions regarding capacity.

Finally, our study was conducted under the following simplifying assumptions: (i) transition probability matrices are known, (ii) capacity adjustment lead times are zero, and (iii) the system features a single, homogeneous capacity type. These assumptions facilitated structural characterization of the optimal policies but limit the external validity and practical generalizability of the results. Future research could relax these restrictions by (a) statistically estimating or learning the transition dynamics from data (through model-free reinforcement learning); (b) incorporating strictly positive and stochastic lead times, including ramp-up and ramp-down dynamics together with potentially nonconvex capacity adjustment costs; and (c) extending the model to multi-resource systems (e.g., multiple nonidentical resources) with explicit coupling constraints across resources. Moreover, empirical calibration using industry data and data-driven policy optimization or evaluation would allow a more rigorous assessment of the managerial relevance and performance of the proposed decision rules.