Dynamic Optimization of Multi-Echelon Supply Chain Inventory Policies Under Disruptive Scenarios: A Deep Reinforcement Learning Approach

Abstract

Addressing two types of supply chain disruptions—frequent short-duration disruptions (e.g., minor natural disasters) and infrequent long-duration disruptions (e.g., geopolitical conflicts, public health crises)—while considering their impact on logistics capacity, this paper proposes a multi-echelon inventory management optimization framework based on the Proximal Policy Optimization (PPO) algorithm. Unlike traditional inventory control models with simplistic assumptions, this study integrates factors such as the frequency, duration, and impact of disruptions into the inventory optimization process. It is designed to coordinate replenishment decisions at the warehouse while reacting to local retailer states. Since retailers share the same cost parameters and demand dynamics, their decision problems are structurally symmetric, which allows us to use a shared policy across retailers and thus keep the learning model compact and scalable. Numerical experiments compare the PPO policy with classical inventory heuristics under various network sizes and disruption types. The results show that PPO consistently achieves lower total costs than the benchmarks, and its relative advantage becomes more pronounced under severe or longer disruptions. These findings suggest that modern policy-gradient methods, combined with simple forms of structural symmetry, can provide an effective and scalable tool for managing disrupted multi-echelon supply chains.

1. Introduction

With the acceleration of globalization and the growing complexity of supply chain networks, supply chains have become indispensable to the stability of the global economy. However, supply chain disruptions (SCDs)—defined as unexpected interruptions or severe delays in the flow of goods, services, or information caused by unforeseen events—are occurring with increasing frequency, posing significant threats to the operational continuity of enterprises. Short-term, high-frequency disruptions, such as earthquakes, floods, and hurricanes, can directly damage logistics infrastructure, leading to production halts, inventory losses, and transportation breakdowns. In contrast, long-term, low-frequency disruptions—such as geopolitical conflicts and abrupt policy changes (e.g., trade disputes, wars, sanctions)—may disrupt cross-border flows of critical raw materials and finished goods, creating severe supply risks and cost escalations. Public health crises, exemplified by the COVID-19 pandemic, exacerbate these risks by causing labor shortages, factory shutdowns, and paralysis of logistics systems, ultimately leading to widespread supply shortages and volatile demand patterns. According to the International Maritime Organization (IMO), global freight costs surged by approximately 20% in 2020 due to port delays. In this context of heightened uncertainty, effective inventory management and improved supply chain resilience—enabling flexible responses to disruptions—have become crucial for safeguarding operational stability and supply chain security. Therefore, there is a clear practical necessity to revisit multi-echelon inventory policies under disruption risk: firms require decision rules that explicitly account for different disruption regimes and remain effective as the network scales, rather than relying on stationary safety-stock rules calibrated ex ante.

Classical inventory models such as the Economic Order Quantity (EOQ) model, the (s, S) policy, and base-stock policies provide important structural insights and have been widely used in practice, but they are typically developed for deterministic or mildly stochastic environments and rely on fixed parameters and stationary assumptions [1,2]. When supply and demand become highly volatile, and disruptions are frequent, especially in multi-echelon networks where upstream and downstream uncertainties interact, such static policies may no longer be effective, which has motivated the use of reinforcement learning (RL) to learn adaptive inventory policies from data [3]. However, most disruption-aware and DRL-based studies still model supply states in a simplified way and focus on single-node or two-echelon settings, so the scalability of RL policies to multi-echelon networks and the explicit role of heterogeneous disruption regimes in shaping inventory decisions remain underexplored [4,5].

To address the above challenges, this study is guided by the following research questions. RQ1: How do heterogeneous disruption patterns—distinguishing frequent short-duration shocks from infrequent long-duration events—affect multi-echelon inventory dynamics when both upstream arrival rates and downstream fulfillment rates are impacted? RQ2: Can a PPO-based replenishment policy coordinate multiple retailers more effectively than classical (s, S) and (S, T) policies in terms of total cost and resilience under different disruption regimes and network scales? RQ3: What managerial guidelines can be derived regarding safety stock configuration and replenishment timing for firms operating multi-echelon networks under disruption risk?

For addressing these questions, this paper studies a multi-echelon inventory system consisting of a central warehouse and multiple retailer nodes, where supply chain disruptions simultaneously affect the effective arrival ratio from the warehouse to retailers and the fraction of goods that can be successfully delivered to customers. Retailers make replenishment decisions in each period based on inventory levels, in-transit orders, realized demand, and disruption status. The problem is formulated as a Markov Decision Process (MDP), in which the state captures multi-echelon inventory and disruption information, actions correspond to continuous order quantities for each retailer, and one-period costs include holding, lost-sales, and logistics/disruption-related components. A PPO-based policy is then trained to minimize the discounted total cost of the overall system and is benchmarked against classical (s, S) and (S, T) policies.

The main contributions of this paper are threefold. First, we develop a unified modeling framework that integrates heterogeneous disruption regimes with multi-echelon inventory dynamics, explicitly allowing disruptions to simultaneously reduce upstream arrival ratios and downstream fulfillment fractions. This extends existing multi-echelon DRL frameworks by providing a richer representation of disruption heterogeneity and its effect on inventory decisions. Second, we employ PPO to learn a single policy that exploits structural symmetry among retailers to coordinate replenishment decisions across nodes, and we systematically compare its performance with classical (s, S) and (S, T) policies. Third, through numerical experiments, we show that PPO achieves lower total cost and higher resilience, especially under infrequent long disruptions and in larger networks, and we derive managerial implications: managers should proactively build higher pre-disruption safety stocks when facing long disruptions, and DRL-based replenishment policies such as PPO can serve as scalable decision-support tools that reduce manual parameter tuning and enhance operational robustness in practice.

The remainder of this paper is organized as follows: Section 2 reviews related literature on inventory management and RL applications in this field. Section 3 presents the dynamic modeling framework for the multi-echelon inventory system under disruptions. Section 4 formulates the MDP and details the PPO-based policy learning process. Section 5 reports experimental results comparing PPO with traditional policies. Section 6 concludes the paper and outlines future research directions.

2. Literature Review

2.1. Research on Inventory Management Problems

Inventory management, as a core component of supply chain management, has been extensively studied over the decades. Early work focused on deriving optimal policies under stylized conditions. The Economic Order Quantity (EOQ) model pioneered by Arrow, Harris, and Marschak [6] established the fundamental cost trade-off between ordering and holding costs, while the (s, S) policy formalized by Arrow [7] was shown to be structurally optimal for periodic review systems with stochastic demand and fixed ordering costs. Clark [8] further demonstrated the optimality of base-stock policies in serial systems with linear holding and backorder costs. Subsequent research extended these ideas to multi-echelon settings, for example, through inter-echelon cost transfer mechanisms [9] and dual-supplier structures with delayed ordering [10], laying the foundation for coordinated control in distribution networks.

As global supply chains became more complex and vulnerable, the literature began to explicitly incorporate disruption risks. Parlar and Berkin [11] introduced a two-state (normal/disrupted) supplier model extending EOQ to supply disruption scenarios with lost sales, and Parlar [12] developed semi-Markov models for ON/OFF supply processes. Schmitt [13] used discrete-time Markov chains to analyze base-stock (S, T) policies under supply disruptions, while Saithong [14] and Taleizadeh [15] refined (S, T)-type models by considering continuous disruption durations and optimizing base-stock levels. Other studies highlighted the multi-dimensional impact of disruptions on lead times and flows: Song and Zipkin [16] and Gupta [17] examined variable lead times and their interaction with Poisson demand; Saputro, Figueira, and Almada-Lobo [18] showed that transportation disruptions amplify inventory fluctuations; Pathy and Rahimian [19] emphasized the role of delivery failures and inventory losses in healthcare supply chains. These works collectively demonstrate that disruptions affect both upstream arrival processes and downstream service levels, but they typically rely on relatively low-dimensional, often binary, disruption representations.

Recent studies have further enriched the modeling of uncertainty using confidence-level and game-theoretic approaches, and Liu [20] considers the impact of cost uncertainty at different confidence levels on supply chain competition. De Giovanni [21] developed a dynamic supply chain game with vertical coordination and horizontal competition, showing how contractual mechanisms and strategic interactions shape inventory and pricing decisions over time. Gao and Hua [22] analyzed a green e-commerce supply chain with delivery-time-dependent demand and epistemic uncertainty, using confidence levels to shape robust decisions on pricing and carbon-emission efforts. Liu [23] introduced the Merton Jump Diffusion (MJD) model to simulate non-stationary demand fluctuations induced by disruptions. Compared with these studies, which focus on strategic or contract-level decisions under epistemic or cost uncertainty, our work concentrates on operational multi-echelon inventory control under stochastic disruption processes and uses DRL rather than analytical game equilibria to handle high-dimensional, state-dependent decisions.

In summary, the classical and disruption-aware inventory literature provides rich insights into optimal policies and risk mechanisms, but most models either rely on simplified two-state disruption structures or treat uncertainty via confidence levels and game-theoretic parameters. They do not explicitly represent heterogeneous disruption regimes (e.g., frequent short vs. infrequent long events) acting simultaneously on upstream arrival ratios and downstream fulfillment, nor do they leverage learning-based methods to adapt inventory policies in such environments.

2.2. Application of Deep Reinforcement Learning in Inventory Management

In parallel, Deep Reinforcement Learning (DRL) has shown strong potential for addressing complex inventory and supply chain problems. At the single-echelon level, Dittrich and Fohlmeister [24] applied DQN-type algorithms to inventory control under demand volatility, while Barat et al. [25] used actor–critic methods for joint inventory–transportation decisions. DRL has also been deployed for domain-specific applications such as perishable inventory management [26], shortage risk mitigation for critical medical supplies [27], and vendor-managed inventory in the semiconductor industry [28]. In the broader supply chain context, Aboutorabet al. [29] used DRL to identify disruption risks, while Zhao et al. [30] showed that DRL can enhance supply chain resilience against external shocks.

For multi-echelon systems, several recent works are particularly relevant. Geevers [31] designed DRL models for continuous-action replenishment in multi-echelon networks. Harsha et al. [32] integrated DRL with mathematical programming to coordinate inventory decisions, and Wang and Lin [33] explored multi-agent RL for collaborative decision-making. More recently, Meisheri [34] proposed a multi-agent DRL framework for multi-echelon inventory management, demonstrating the feasibility of learning coordinated policies across echelons. Babazadeh [4] developed a hybrid ANN–MILP model for agile recovery production planning for PPE under sharp demand spikes, while Wang [35] introduced a DRL-based dynamic replenishment approach for multi-echelon inventory systems. These studies confirm that DRL and related machine-learning techniques are powerful tools for multi-echelon and disruption-prone environments.

At the methodological level, the literature has compared different DRL algorithms and architectures. Foundational DQN variants are widely used for discrete decisions [36], whereas actor–critic methods, including A2C and other policy-gradient algorithms, are preferred for more complex or continuous control tasks [37]. Hybrid approaches combine DRL with mathematical programming [32], discrete-event simulation [38], or reward-shaping and architecture tuning [26]. However, most DRL applications either consider generic demand or cost uncertainty or treat disruptions implicitly, and many are limited to single-node or simple two-echelon structures without explicitly modeling heterogeneous disruption regimes.

2.3. Positioning and Methodological Novelty of This Study

Against this background, the studies reviewed in Section 2.1 and Section 2.2 can be viewed as the current state-of-the-art along two closely related strands. Our work contributes to the literature in two main directions. First, relative to classical and confidence-level-based disruption models [13,39], we propose a unified multi-echelon modeling framework that explicitly captures disruption heterogeneity along two dimensions—frequency and duration—and allows disruptions to simultaneously reduce the effective arrival ratio from the warehouse to retailers and the fraction of goods delivered to customers. This extends existing disruption modeling, which typically uses two-state or aggregate risk parameters, by embedding a richer Markovian disruption structure directly into the inventory dynamics and cost components.

Second, compared with current multi-echelon DRL frameworks [40,41,42,43], our study emphasizes the methodological integration of disruption heterogeneity and DRL. We formulate the problem as a high-dimensional MDP where the state includes multi-echelon inventories, in-transit orders, and disruption states, and the action space consists of continuous order quantities for multiple retailers. We adopt Proximal Policy Optimization (PPO) as the learning algorithm, leveraging its stable policy-gradient updates for long-horizon, non-stationary reward structures in disruption-prone systems, and show how a single PPO agent can exploit structural symmetry among retailers to coordinate replenishment decisions across nodes. Numerical experiments systematically compare the learned PPO policy with classical (s, S) and (S, T) policies under different disruption regimes and network scales, demonstrating not only cost advantages but also resilience improvements. In this sense, the methodological superiority of our approach lies in jointly modeling disruption heterogeneity, multi-echelon interactions, and continuous-action DRL within a coherent optimization framework.

3. Modeling the Multi-Echelon Inventory Problem Under Supply Chain Disruptions

3.1. Problem Description

This study focuses on managing multi-echelon inventory systems within supply chains subject to disruptions, as shown in Figure 1. The core system under investigation comprises a central warehouse supplying a network of M retailer nodes. System operation unfolds over discrete time periods (e.g., weeks). During each period, every retailer node must make real-time, online replenishment quantity decisions. These decisions are based on the prevailing inventory level, observed market demand, and the overall state of the supply chain, with particular emphasis on its disruption status.

Figure 1. Illustration of a multi-level supply chain.

To quantify the specific impact of disruptions on logistics capacity, this study refers to the framework of Pathy et al. [19] and introduces a key disruption-loss coefficient mechanism. The actual quantity of goods received by retailers is only a portion of their expected arrivals and is adjusted proportionally by ${\mu }_1\in \left[\mathrm{0,1}\right]$. Meanwhile, during the delivery process to the final customers, not all available inventory can be delivered intact. Only a proportion ${\mu }_2\in \left[\mathrm{0,1}\right]$ of the goods can remain intact and be delivered to customers.

3.2. Premises and Assumptions

To accurately capture the inherent dynamics and uncertainties of multi-echelon inventory management and ensure model robustness and adherence to practical business logic, this study adopts the following key assumptions:

(1)

The central warehouse possesses infinite storage capacity, whereas each retailer node operates under a finite storage capacity constraint, ensuring actual inventory levels never exceed prescribed maximum limits.

(2)

Replenishment lead time is fixed and deterministic, constituting an integer multiple of the periodic review interval; retailers employ a cycle-based inventory review policy triggered at each discrete time period.

(3)

Retailer nodes exhibit operational independence: each maintains a dedicated, isolated supply chain link and internal inventory; surplus inventory cannot be shared across nodes, and replenishment decisions are based solely on local inventory levels, in-transit orders, and realized customer demand.

(4)

Customer demand per retailer per period is stochastic, modeled as a Poisson-distributed random variable; demand occurrences are mutually independent across retailers and time periods, and the distribution mean/variance parameters are dynamically adjustable to reflect heterogeneous market conditions.

(5)

If the inventory is sufficient to meet demand, all remaining inventory will be carried over to the next period, which results in permanent sales loss.

(6)

Replenishment order quantities are continuous non-negative real numbers, ensuring practical implementability.

(7)

Supply chain disruptions manifest under two archetypal modes: frequent-short events and rare-prolonged events; these may occur at critical stages (e.g., warehouse supply, transport links, retailer operations), significantly degrading associated logistics capabilities.

(8)

When the system enters a disruption state, the changes in ${\mathrm{\mu }}_1$ and ${\mathrm{\mu }}_2$ apply simultaneously to all retailers, rather than independent local disruptions at individual retailers.

These assumptions are consistent with mainstream models of multi-echelon inventory control and supply chain disruptions in the OR/MS literature. They abstract away from some operational details to keep the problem analytically and computationally tractable, while still reflecting key features of real distribution systems.

3.3. Definitions Related to the Issues

Table 1 lists the symbols and their explanations used for problem formulation. In inventory management problems, adjusting inventory levels based on factors such as current inventory, demand, replenishment quantity, and supply chain status is central. The inventory update rules are as follows:

Table 1. Mathematical symbols and their meanings.

Symbol	Explanation
$m$	Retailer ID
$\tau$	Time period index
$L$	Lead time of retailers
${\mu }_1$	Percentage of goods from the warehouse to the retailer
${\mu }_2$	Percentage of goods from the retailer to the customer
${IL}_m\left(\tau \right)$	Inventory level of the retailer at the end of $\tau$ period
${il}_m\left(\tau \right)$	Initial inventory of $\tau$ period
$D_m\left(\tau \right)$	Customer demand faced by the retailer at $\tau$
${\widehat{IO}}_m\left(\tau \right)$	$\mathrm{L}\mathrm{i}\mathrm{s}\mathrm{t} \mathrm{o}\mathrm{f} \mathrm{a}\mathrm{l}\mathrm{l} \mathrm{o}\mathrm{u}\mathrm{t}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{o}\mathrm{r}\mathrm{d}\mathrm{e}\mathrm{r}\mathrm{s}$ at $\tau$
${io}_m\left(\tau \right)$	Elements in the list, indicating the expected arrival at $\tau$
$Et\left(\tau \right)$	Interrupt situation at $\tau$
$O_m\left(\tau \right)$	Orders sent at $\tau$
$q_{ m}^{LS}\left(\tau \right)$	The quantity of sales loss
$q_{ m}^{Loss}\left(\tau \right)$	The quantity of logistics loss
${MaxIL}_m$	Maximum inventory level
${ls}_m$	Unit cost of sales loss
${loss}_m$	Unit cost of logistics loss
$f_m$	Contract cost
$h_m$	Unit holding cost

Receiving Inventory: ${il}_m\left(\tau \right)$ and ${IL}_m\left(\tau \right)$ are used to define the initial inventory and ending inventory of a retailer $m\in M$ in period $\tau \in [1,T]$ at the beginning of the period, respectively. If there is an in-transit order arriving within the period $\tau \in [1,T]$, then the initial inventory of the retailer $\mathrm{m}\in \mathrm{M}$ is

$${il}_m\left(\tau \right)={IL}_m\left(\tau -1\right)+{io}_m\left(\tau \right)$$

(1)

If a disruption occurs, it is

$${{il}_m\left(\tau \right)=IL}_m\left(\tau -1\right)+{\mu }_1{io}_m\left(\tau \right)$$

(2)

Updating Inventory Facing Customer Demand: $D_m\left(\tau \right)$ is used to represent the actual demand of the retailer $m\in M$ for the commodity in the period $\tau \in [1,T]$. $D_m$ is stochastic and follows a Poisson distribution. The ending inventory calculation formula is

$${IL}_m\left(\tau \right)=\mathrm{m}\mathrm{a}\mathrm{x}(0,il-Dm\left(\tau \right))$$

(3)

Loss Measurement: $q_{ m}^{LS}\left(\tau \right)$ and $q_{ m}^{Loss}\left(\tau \right)$ are used to represent the sales loss and logistics loss quantity that occurred for the retailer $m\in M$ in the period $\tau \in [1,T]$, respectively. Sales loss occurs when supply falls short of demand; logistics loss only occurs in the event of a disruption. The calculation formulas are

$$q_{ m}^{LS}\left(\tau \right)=\mathrm{m}\mathrm{a}\mathrm{x}(0,Dm\left(\tau \right)-il)$$

(4)

$${ q}_{ m}^{Loss}\left(\tau \right)=\left(1-{\mu }_1\right)\ast {io}_m\left(\tau \right)+\left(1-{\mu }_2\right)\ast \mathrm{m}\mathrm{i}\mathrm{n}(\mathrm{D}\mathrm{m}\left(\mathrm{\tau }\right),{\mathrm{i}\mathrm{l}}_{\mathrm{m}}(\mathrm{\tau }\left)\right)$$

(5)

In-Transit Order Management: After meeting the current demand at $\tau$, retailer $m\in M$ needs to make the next order decision based on the inventory status. This can be represented as $O_m\left(\tau \right)$ after the order is dispatched, it is added to the in-transit list and arrives at $\tau +L_m$.

3.4. Cost Composition and Optimization Objective

3.4.1. Cost Composition

In this paper, the costs incurred by a retailer in each time period mainly include contract fees, inventory costs, sales loss fees, and goods loss fees. Specific definitions are as follows:

(1)

Contract fees ${\mathrm{F}}_{\mathrm{m}}(\mathrm{\tau })$ are the agreement costs between the retailer and the supplier, including the fees for placing orders and the fixed fees stipulated in the transportation agreement. In this paper, the contract fee is a fixed amount determined by mutual agreement, and unit ordering costs are not considered.

$$F_m(\tau )=\left\{\begin{array}{c}{f}_m order \\ 0 no order \end{array}\right.$$

(6)

where ${\mathrm{f}}_{\mathrm{m}}$ is the contract fee for retailer m ordering from the supplier.

(2)

Inventory costs $H_m\left(\tau \right)$ refer to the expenses the retailer must bear to maintain a certain inventory level, including inventory-holding costs and warehousing costs.

$$H_m\left(\tau \right)=h_m·{IL}_m\left(\tau \right)$$

(7)

where $h_m$ is unit inventory cost.

(3)

Sales loss fees ${LS}_m\left(\tau \right)$ occur when demand exceeds the current inventory, leading to an inability to meet customer requirements. In inventory management, sales loss fees reflect the negative impact of the inventory strategy’s failure to respond effectively to demand fluctuations and supply chain disruptions.

$${LS}_m\left(\tau \right)={ls}_m·q_{ m}^{LS}\left(\tau \right)$$

(8)

where ${ls}_m$ is the unit sales loss cost.

(4)

Goods loss ${Loss}_m\left(\tau \right)$ fees refer to the cost of commodity losses caused by reasons such as transportation disruptions, improper warehouse management, or supply chain disruptions. Especially under supply chain disruption scenarios, transportation delays and damage to goods will lead to increased losses.

$${Loss}_m(\tau )={{loss}_m·\mathrm{q}}_{ m}^{Loss}\left(\tau \right)$$

(9)

where ${loss}_m$ is the unit cost of goods loss.

3.4.2. Optimization Objective

By accurately calculating and optimizing these cost items, the goal is to achieve an optimal inventory management strategy and reduce the overall cost of the supply chain system. These costs can be calculated as follows:

Within $\tau$ period, the total cost of the retailer $M$ can be expressed as follows:

$$cost\left(\tau \right)=f_m\left(\tau \right)+H_m\left(\tau \right)+LSm\left(\tau \right)+{Loss}_m\left(\tau \right)$$

(10)

The main objective of this paper is to design and optimize multilevel inventory management strategies in a complex, volatile, and uncertain supply chain environment, aiming to minimize the long-term total operating cost of the system. Uncertainty in the system stems from two primary sources: demand volatility, characterized by randomness and the difficulty of accurately forecasting retailer demand; and supply chain disruptions, which probabilistically reduce goods arrivals and delivery rates. Together, these factors render the cost structure highly dynamic and difficult to predict. The objective function is

$$\begin{gathered} \underset{\tau \in \left[1,T\right],\mathit{m}\in M}{\mathit{min}}\sum _{\tau =1}^{T}cost\left(\tau \right)=\underset{\tau \in \left[1,T\right],\mathit{m}\in M}{\mathit{min}}\sum _{\tau =1}^T{f}_m\left(\tau \right)+\underset{\tau \in \left[1,T\right],\mathit{m}\in M}{\mathit{min}}\sum _{\tau =1}^T{H}_m\left(\tau \right) \\ +\underset{\tau \in \left[1,T\right],\mathit{m}\in M}{\mathit{min}}\sum _{\tau =1}^{T}LSm\left(\tau \right)+\underset{\tau \in \left[1,T\right],\mathit{m}\in M}{\mathit{min}}\sum _{\tau =1}^T{Loss}_m\left(\tau \right) \end{gathered}$$

(11)

4. PPO-Based Multilevel Inventory Strategy Optimization

Traditional inventory methods, such as the Economic Order Quantity (EOQ) model and (s, S) policies, work in stable environments but are inadequate under frequent disruptions and demand volatility due to their fixed assumptions. This paper employs a Deep Reinforcement Learning (DRL) approach and Proximal Policy Optimization (PPO), which learns adaptive policies through agent-to-environment interaction. PPO offers greater flexibility and robustness, stabilizes training via a clipping mechanism, avoids gradient issues, and efficiently handles continuous action spaces. These features make PPO well-suited for dynamic, uncertain, multi-level inventory management, enabling adaptive strategy adjustments in complex supply chains.

4.1. Markov Decision Process Model

To better solve the optimization problem, this paper first models the inventory management problem as a Markov Decision Process (MDP). By defining states, actions, and rewards, the MDP provides structured support for inventory decisions, enabling them to cope with different scenarios of supply chain disruptions and demand changes. As illustrated in the figure, the specific decision-making process at each time step is as follows:

The agent observes the environment $S_{\tau }$. The agent first perceives the environment by observing the current state $S_{\tau }$. The state describes all the necessary information about the system at $\tau$, including the current inventory levels of retailers, in-transit orders, market demand, and supply chain disruption status.

The agent selects an action $a_{\tau }$. Based on the observed $S_{\tau }$, the agent selects an action according to its policy $\mathrm{\pi }$. The action represents the behavior the agent can take in the current state ${ S}_{\tau }$. In the problem structure of this paper, the action refers to the decision made by retailers at a specific point in time. Specifically, it is each retailer deciding how many orders to place for the next time period based on their current inventory status and demand forecast.

The environment transitions to a new state, and after the action occurs, the system transitions from $S_{\tau }$ to $S_{\tau +1}$. The transition triggers a reward. $R_{\tau }$ reflects the quality of the current decision. Lower inventory holding costs, fewer sales losses, and smaller logistics costs typically correspond to higher rewards (a smaller negative total cost).

The goal of the MDP is to find a policy minimizing the infinite sum of discounted costs, where a decision is a mapping from states to actions. Note that the objective of this paper is to minimize the long-term average cost. This means that, in practice, $\tau$ is chosen sufficiently high to reflect the infinite-time cost. This objective can therefore be expressed as shown in the equation:

$${\pi }^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g} \underset{\pi \in \mathit{\Pi }}{\mathit{minE}}[\sum _{\tau =0}^{\infty }{\gamma }^{\tau }·c_{\tau }\left(S_{\tau },a_{\tau }^{\pi }\right)]$$

(12)

where $\gamma$ is the discount factor balancing the influence of future and current rewards. $a_{\tau }^{\pi }$ denotes the action taken by the policy $\pi$ at $\tau$.

The multilevel inventory management problem is one that requires multi-step, continuous decision-making. At each time step, the agent outputs the optimal decision, eventually forming a complete service strategy. The decision-making process at each time step is modeled as a Markov Decision Process.

4.2. Definition of States, Actions, and Rewards

MDPs can be defined as a tuple $(S,A,R)$, where S represents the set of environment states, $A$ represents the set of actions available for the agent to choose, and R represents the set of reward values provided by the environment after each decision.

State: Includes the current inventory level of the retailer, outstanding orders, market demand, and the status of supply chain disruptions.

$$S_{\tau }=\left[{IL}_m\left(\tau \right),D_m\left(\tau \right),{\widehat{IO}}_m\left(\tau \right),Et\left(\tau \right)\right]$$

(13)

where ${IL}_m\left(\tau \right)\in 0,{MaxIL}_m$. To track the replenishment process, an in-transit inventory list ${\widehat{IO}}_m\left(\tau \right)$ is established. After a certain time, the order information will be recorded as ${io}_m\left(\tau \right)$ in the in-transit inventory list at $\tau$ position, and once the goods arrive in the inventory (the goods ordered at $(\tau -L$), they will be removed from the in-transit list. $Et\left(\tau \right)=0 or 1$, represents whether there is an interruption.

Action: The action space is defined as the replenishment decision of each retailer at every decision epoch. Specifically, each retailer determines its order quantity for the next period based on the current state. Since the replenishment quantity is a continuous, non-negative variable in practice, it is discretized to form a finite action set. The continuous order quantity is first normalized into the interval $\left[\mathrm{0,1}\right],$ then discretized into evenly spaced grid points (e.g., rounded to two decimal places). This discretization ensures that the decision space remains computationally tractable while preserving the ability to approximate continuous order decisions.

$$\begin{array}{ccc}A& =& \{O_1\left(\tau \right),O_2\left(\tau \right)\dots O_m\left(\tau \right)\}\end{array}$$

(14)

After the agent selects an action, the action value is mapped to $O_m\left(\tau \right)$ to obtain ${\mathrm{a}}_{\mathrm{\tau }}$, that is, $a_{\tau }=\left[O_m\left(\tau \right)\right] (O_m\left(\tau \right)\in \left(0,{MaxIL}_m\right))$ This processing method not only ensures that the values related to the order quantity do not exhibit excessive fluctuations, which better meets the requirements of the actual business scenario.

Reward: In the inventory problem considered in this paper, the environment is cost-minimizing, and the reward fed back to the agent is a negative cost rather than a normalized penalty. Specifically, at each decision epoch $\mathrm{\tau }$, we first compute the total cost $C_{\tau }$ as the sum of holding cost, shortage (lost-sales) cost, and logistics/disruption-related cost. The immediate reward is then defined as follows:

$$r_{\tau }=-C_{\tau }$$

(15)

Because all cost components are measured in the same monetary unit and the per-period cost magnitudes are moderate (see Table 2), we do not apply additional normalization or clipping to $r_{\tau }$. This linear transformation preserves the optimal policy while keeping the scale of rewards within a numerically stable range for PPO training.

Table 2. Retailers’ parameter settings.

Symbol	Retailer 1	Retailer 2	Retailer 3	Retailer 4
${Il}_o$	50	100	150	200
$f_m$	10	20	30	50
$h_m$	5	10	15	20
${ls}_m$	25	50	75	100
${loss}_m$	80	120	200	300
${MaxIL}_m$	200	450	700	1000

4.3. PPO Method Model Framework

After modeling the multi-echelon inventory management problem as a Markov Decision Process (MDP), this paper adopts the Proximal Policy Optimization (PPO) algorithm to optimize retailer replenishment decisions under disruption scenarios. PPO, a policy-gradient method proposed by Schulman et al. (2017) [44] within an actor–critic framework, stabilizes training via a clipping mechanism, improving convergence and robustness in dynamic environments. It achieves high sample efficiency through multiple mini-batch updates, reducing computational costs in simulations. PPO also offers strong flexibility and scalability, adapting to environmental changes and enabling efficient multi-node parallel decision-making, thus providing effective algorithmic support for complex supply chain inventory optimization.

The PPO algorithm training process is illustrated in Figure 2. At each time period $\mathrm{\tau }$, the agent first observes the environmental state $S_{\tau }$. Based on the current state, the policy network ${\pi }_{\theta }\left(a\right|s_{\tau })$ outputs a parameterized distribution over actions. By stochastically sampling from this distribution, the agent obtains the actual order quantity decision $a_{\tau }$ to execute. After executing the action, the agent receives an immediate reward $R_{\tau }$ (i.e., the negative total periodic cost) and the next state ${\mathrm{S}}_{\mathrm{\tau }+1}$ from the environment. To evaluate the relative advantage of each action under the current policy, the Generalized Advantage Estimation (GAE) method is employed. The advantage value $\widehat{A_t}$ is calculated as follows:

$${\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)$$

(16)

$$\widehat{A_t}=\sum _{l=0}^{T-t} {(\gamma \lambda )}^l{\delta }_{t+l}$$

(17)

Figure 2. Training flowchart of the PPO algorithm.

Through GAE, the improvement potential of each action can be efficiently estimated while controlling variance, providing the basis for subsequent policy updates. Based on the collected trajectory data and computed advantage estimates, the PPO updates the policy network parameters $\theta$ by minimizing a clipped objective function to stably improve policy performance. The optimization objective is

$$L^{CLIP}\left(\theta \right)=E_t\left[min\left(r_t\left(\theta \right)A_t,clip\left(r_t\left(\theta \right),1-ϵ,1+ϵ\right)A_t\right)\right]$$

(18)

Simultaneously, the value network parameters $\varphi$ are updated by minimizing the mean squared error loss to enhance the accuracy of value estimation. The training process utilizes mini-batch Stochastic Gradient Descent (SGD), performing multiple rounds of updates per batch of samples.

5. Experimental Setup

5.1. Simulation Parameter Settings

In inventory management, the problem scale, defined by the number of retailers, demand volatility, inventory capacity, and cost coefficients, directly affects decision complexity and optimization outcomes. Multi-echelon systems amplify these challenges due to increased coordination needs and resource allocation complexity. Demand and capacity fluctuations further impact stability and efficiency, heightening uncertainty. To assess the PPO performance across different scales, this study configures simulation parameters to reflect disparities between small and large retailers, as shown in Table 2.

Supply Interruption Modeling: To accurately simulate disruption scenarios triggered by unforeseen events within a supply chain, this paper adopts a two-state supply process for modeling. Snyder et al. (2016) [45] indicate that this process can effectively describe a wide range of interruption scenarios, spanning from frequent but brief disruptions to infrequent yet prolonged ones. The parameter α denotes the probability of the system transitioning from the normal state to the disrupted state, while the parameter β represents the probability of the system recovering from the disrupted state back to the normal state. The average duration in each state follows a geometric distribution. Within the stochastic process, the probability of the system being disrupted for exactly $\tau$ review periods is given ${\displaystyle \frac{\alpha }{\alpha +\beta }}$ [45]. Consequently, the average duration of the disrupted state can be inferred as ${\displaystyle \frac{\alpha }{\alpha +\beta }}$, and the average duration of the normal state as $P=1-{\displaystyle \frac{\beta }{\alpha +\beta }}$. The interruption parameters utilized in this paper are adopted from the settings defined by Czerniak et al. [46], as shown in Table 3.

Table 3. Disruption type and probability.

Interrupt Type	$\mathbf{Interruption} \mathbf{Probability} \mathit{\alpha }$	$\mathbf{Recovery} \mathbf{Probability} \mathit{\beta }$
Frequent but short	0.5	0.5
infrequent but long	0.01	0.01

Hyperparameter Settings and Tuning: To select these hyperparameters, we followed a two-stage tuning procedure. First, we started with standard PPO defaults commonly used in the DRL literature and adjusted them to ensure numerically stable training in our multi-echelon inventory environment. Second, we performed a coarse grid search on a representative scenario with two retailers and disruption type 2 (infrequent, long disruptions). We then selected the configuration that achieved the lowest average total cost with smooth, non-divergent learning curves. This final configuration is fixed across all other experimental settings so that performance differences can be attributed to disruption patterns and network scales rather than case-specific hyperparameter retuning. The detailed hyperparameter settings are shown in Table 4.

Table 4. Hyperparameter settings.

Hyperparameter	Value
iterations	10,000
layers	2 hidden layers with (64, 64)
learning rate	linear from 5 × 10−4 to 1 × 10−5
batch size	64
n steps	2048
gamma	0.99
gae lambda	0.95
ent coef	0.01
n epochs	10
clip range (epsilon)	0.2

Implementation details and training stability: We used a shared actor–critic multilayer perceptron with two hidden layers of 64 units and ReLU activations, optimized by Adam. The state variables and costs were scaled to moderate ranges, and no additional batch or layer normalization was applied. PPO was trained for a fixed maximum of 10,000 iterations; during development, we monitored the moving average of the discounted total cost and observed that learning curves plateaued well before this limit, so we did not employ early stopping. For reproducibility, we used a fixed global random seed for both the simulation environment and the PPO implementation, and we applied the same seed to all compared policies when computing the costs in Table 5, Table 6 and Table 7. We also tracked the PPO clipped objective, value-function loss, and policy entropy throughout training and did not observe divergence or collapse, indicating stable training behavior.

Table 5. Comparison of the results.

m	Retailers	Interrupt Type	(s, S)	(S, T)	Ours
2	1, 4	1	35,790	36,300	35,000
	1, 4	2	34,000	34,500	31,500
3	1, 2, 4	1	54,302	55,020	51,500
	1, 2, 4	2	47,302	48,200	46,850
4	1, 2, 3, 4	1	74,108	76,090	70,505
	1, 2, 3, 4	2	63,108	64,200	62,300

Table 6. Parameter sensitivity analysis.

The Influencing Factors	Represent	The Sensitivity Analysis
Arrival rate from warehouse to retailer ${\mu }_1$	0.7	0, 0.5, 0.7, 1
Retailer to customer goods arrival rate ${\mu }_2$	0.8	0, 0.5, 0.8, 1
lead time $L$	2	2, 3, 4

Table 7. Impact of lead time L on the retailer’s cost.

L	m	Interrupt Type	(s, S)	(S, T)	Ours
3	2	1	40,000	41,000	36,050
		2	34,000	35,000	32,250
	3	1	58,000	60,000	54,000
		2	53,000	54,090	50,000
	4	1	77,000	80,000	73,800
		2	67,000	68,200	65,000
4	2	1	40,000	42,000	38,550
		2	36,000	38,000	33,500
	3	1	68,000	70,000	59,000
		2	55,000	59,000	52,000
	4	1	90,000	92,980	79,000
		2	74,000	79,000	68,000

5.2. Experimental Results Analysis

5.2.1. Comparative Experiments on Effectiveness

The (s, S) and (S, T) inventory policies are employed as baselines to evaluate the algorithm’s performance. For the effectiveness experiments, scenarios with two, three, and four independent retailers and a lead time of two were considered. Given that the retailers are mutually independent and exhibit no synergistic effects, the sum of the locally optimal policies for each retailer can be approximated as the global optimum in the absence of cross-node coordination [7]. Consequently, the baseline policies are constructed from the optimal policy for each individual retailer, determined using a grid-search method within the search range $s\in \left(0,{MaxIL}_m\right),S\in \left(0,{MaxIL}_m\right)$. The average total cost over a 50-time-step horizon was computed via Monte Carlo simulation for each candidate (s, S) pair within the grid space; the optimal (s, S) combination was then selected as the solution yielding the minimum cost. An identical optimization procedure was employed for the (S, T) inventory policy.

The results shown in Figure 3 and Figure 4 are obtained when only two retailers are considered. It is clear that the PPO algorithm outperforms the other two baseline algorithms, especially in the face of infrequent and long-duration interruptions. Specifically, in the case of long and infrequent disruptions, the PPO algorithm can adjust the inventory policy more effectively, reduce the overall cost of the supply chain by 8%, and improve the resilience of the system. This shows that the PPO algorithm has stronger adaptability and optimization ability when dealing with longer supply chain disruptions. In contrast, in frequent and short-duration outage scenarios, the traditional baseline algorithm can quickly respond and adjust to achieve better performance, and the PPO algorithm can only improve by 2%.

Figure 3. Comparison of algorithms in frequent and short duration interruptions.

Figure 4. Comparison of algorithms in infrequent and long-duration interruptions.

It can be seen from Table 5 that with the increase in the number of retailers (M), the traditional algorithm is unable to cope with the scale expansion of the complex supply chain. Especially when the number of retailers increases, the traditional (S,s) and (S,T) strategies gradually lose efficiency in terms of inventory allocation and coordination. Due to the inability to effectively deal with the increasingly complex dependencies and dynamic changes between supply chain nodes, traditional algorithms often lead to uneven resource allocation and inventory instability, which affects the optimization effect of the overall system.

To better understand these cost improvements, we qualitatively examine the behavior of the learned PPO policy. Under frequent short disruptions (Type 1), PPO behaves close to a base–stock rule: inventory positions are kept around a moderate target, and order quantities fluctuate only mildly. In this regime, the (S, T) policy can also track the shallow shocks reasonably well, which explains why the relative cost advantage of PPO in Figure 3 and Figure 4 remains modest when disruptions are frequent and short.

Under infrequent long disruptions (Type 2), PPO adopts a more conservative, anticipatory behavior. During long normal periods, the agent gradually builds higher pre-disruption inventory than the (s, S) and (S, T) policies, and once a disruption occurs, it reduces both the frequency and size of orders, relying more on positioned stock. This anticipatory buffer pattern reduces sales loss and logistics loss costs during long disruption spells and leads to the more pronounced 8–10% cost reductions reported in Table 5 as the number of retailers increases. Moreover, when the network scales from two to four retailers, PPO exploits the symmetry of the system to coordinate replenishment across nodes and avoid extreme stockouts at individual retailers, while the static policies optimize each retailer in isolation and thus suffer from higher total costs.

5.2.2. Comparative Experiments on Generalizability

To further validate the algorithm’s robustness and performance, this paper conducts comparative experiments on generalization to assess the sensitivity of experimental results to variations in key parameters. The objective of these experiments is to examine the algorithm’s adaptability across diverse scenarios and conditions, ensuring stable operation and effective solutions in various environments. To this end, multiple key parameters were varied, and their impacts on algorithm performance were evaluated by comparing experimental outcomes. The generalization parameters used are detailed in Table 6. Specifically, the experiments encompass two primary components:

Variations in Environmental Uncertainty: Uncertainties inherent in the environment, such as supply chain disruptions, transportation delays, and demand forecast errors, represent common challenges in inventory management. Therefore, differing disruption levels were considered as they might affect algorithm performance. Different disruption severities were characterized by the goods arrival rate. Changes in the goods arrival rate reflect varying degrees of supply chain disruption severity, encompassing the following four scenarios: Complete disruption, Severe disruption, Minor disruption, No disruption. By adjusting parameters ${\mathrm{\mu }}_1$ and ${\mathrm{\mu }}_2$, the impact of these varying disruption severities within the supply chain was simulated. For example, A configuration with ${\mu }_1=0.5$ and ${\mu }_2=0.8$ represents a severe disruption between the supplier and retailer, and a minor disruption between the retailer and customer. The corresponding results are presented in Figure 5 and Figure 6.

Figure 5. Cost comparison under infrequent long-duration disruptions. (a–d) represent ${\mu }_1=0, 0.5, 0.7, 1$, respectively.

Figure 6. Cost comparison under frequent short-duration disruptions; (a–d) represent ${\mu }_1=0, 0.5, 0.7, 1$, respectively.

Experimental results show that the PPO advantage grows with disruption severity. In severe, large-scale, and long-duration events, it effectively adjusts inventory policies, lowering total costs and enhancing stability and adaptability, demonstrating strong flexibility and robustness in complex, uncertain environments. Comparing traditional methods, the (S, T) policy generally outperforms (s, S) under most disruption conditions due to its ability to flexibly adjust order quantities, maintain optimal inventory levels, and minimize losses from overstock or stockouts. In contrast, (s, S) relies on fixed bounds and historical demand, limiting responsiveness to sudden interruptions. However, as disruptions diminish or remain minor, the performance gap narrows, and (s, S) regains the advantage. In stable, predictable environments, it leverages simplicity, efficiency, and lower operational complexity, making it more adaptive and cost-effective in a steady state.

Supply chain parameter variation: In supply chain management problems, different supply chain characteristics can significantly affect the performance of the algorithm. By tuning these parameters, it is possible to analyze how the algorithm adapts to supply chain environments of different sizes and complexities. We then tested the performance difference between the PPO algorithm and the baseline strategy under different lead times, thus assessing its generalization ability under multiple supply chain conditions.

The results shown in Table 7 indicate that the lead time $\left(L\right)$ exerts a relatively limited influence on the performance of the PPO algorithm. The algorithm demonstrates minimal performance variation across different lead time settings. This suggests that the PPO algorithm exhibits robust stability and adaptability when managing supply chain scenarios characterized by varying lead times. In contrast, traditional algorithms often encounter response lag issues when faced with significant changes in lead time. This lag typically leads to a decrease in inventory management efficiency. Unlike these traditional approaches, the PPO algorithm, by leveraging adaptive strategies from Deep Reinforcement Learning, dynamically adjusts inventory management decisions. This capability allows it to effectively optimize synergistic effects between nodes, even in systems with a large number of retailers, thereby enhancing the overall efficiency of the supply chain. Consequently, the PPO algorithm demonstrates superior systemic adaptability and stability, enabling it to effectively balance demand and inventory across various nodes within complex and large-scale supply chain environments.

The sensitivity results in Figure 5 and Figure 6 show how PPO adapts to different disruption severities. As the arrival rates μ₁ and μ₂ decrease, all policies implicitly increase safety buffers, but PPO does so in a more state–dependent way: when disruption risk is high, it orders earlier and slightly more in the normal state, and then tapers orders once the system enters a sustained disruption. The fixed (s, S) and (S, T) rules, calibrated for a nominal environment, cannot dynamically adjust this trade–off, which leads to either pronounced stockouts or over–stocking when disruption severity changes.

A similar mechanism appears when varying the lead time L in Table 7. For longer lead times, PPO learns smoother and earlier replenishment actions to compensate for delayed arrivals, thereby mitigating both lost–sales costs and the need for sudden large orders when inventory becomes critically low. In contrast, the benchmark policies systematically lag behind changes in L because their parameters are fixed ex ante, so their total cost grows faster than that of PPO as L increases. Overall, PPO improves performance by learning disruption–aware policies that adjust safety–stock levels and replenishment timing over a wide range of operating conditions.

6. Conclusions

In this study, we proposed an inventory optimization framework based on the Proximal Policy Optimization (PPO) algorithm for managing multi-echelon supply chains under disruption scenarios. Numerical experiments show that leveraging the underlying structural symmetry of the multi-echelon system enables PPO to generalize policies across retailers, while still adapting to disruption-induced asymmetries in arrival and fulfillment rates.

Our experimental results demonstrate that PPO outperforms traditional inventory management models, particularly in reducing total supply chain costs during disruptions. Specifically, PPO dynamically adjusts replenishment strategies, improving the resilience of the supply chain while minimizing both inventory and sales loss costs. These findings have significant implications, both theoretically and practically.

This research contributes to the literature by applying reinforcement learning, specifically PPO, to the optimization of inventory management in disrupted supply chains. It challenges traditional inventory control models that rely on fixed rules and assumptions, highlighting the potential of adaptive learning methods to better respond to dynamic and uncertain environments. Our study provides valuable insights into how reinforcement learning can be used to optimize decision-making in multi-echelon systems, where conventional methods often struggle to achieve optimal performance.

From a practical standpoint, our study provides actionable insights for supply chain managers dealing with disruptions. PPO’s ability to adapt its replenishment policies in real-time offers a flexible and efficient solution for optimizing inventory management, especially in environments where supply chain disruptions are frequent. By adopting PPO, companies can enhance their decision-making processes, improving both cost efficiency and service levels during periods of uncertainty. This is particularly valuable for industries like retail and manufacturing, where traditional models fall short in handling disruptions, leading to stockouts and excessive inventory costs.

While this study demonstrates the potential of PPO for optimizing supply chain management, there are limitations to our work. Our model assumes simplified disruption types and does not incorporate certain real-world complexities, such as supplier diversity and transportation delays. Additionally, the scalability of the model to larger, multi-echelon supply chains requires further exploration.

Future research could explore more complex disruption scenarios and incorporate additional features such as demand forecasting and real-time data integration. Further studies could also extend the PPO model to multi-retailer and multi-echelon systems, addressing the scalability challenges and improving its practical applicability in real-world supply chains.

In conclusion, this research not only contributes to advancing reinforcement learning techniques in supply chain management but also provides practical solutions for real-world supply chain optimization in disrupted environments. The findings lay the foundation for further exploration of adaptive learning models in supply chains, ultimately helping organizations better navigate uncertainty and enhance operational efficiency.