Abstract
The increasing demand for sustainable development necessitates sophisticated computational frameworks that facilitate intricate, long-term decision-making in circular economy systems. Therefore, we developed a novel machine learning based Markov decision framework that effectively combines predictive analytics with sequential decision theory to provide adaptive and scalable optimization of resource circulation processes. The developed architecture utilizes supervised learning to predict material flow dynamics and exploits the Markov decision process (MDP) to develop reward-sensitive strategies under environmental variability and operational uncertainty. The study results on municipal waste management demonstrate the framework’s superior performance in resource recovery efficiency, policy responsiveness, and sustainability impact compared to traditional static or rule-based methods. Such findings contribute to the development of a generalizable, learning-enabled tool for dynamic multi-stage decision assistance in complex and unpredictable circular economy contexts.
1. Introduction
Attaining a circular economy (CE) is essential for tackling sustainability and resource issues. Nevertheless, CE systems are dynamic and uncertain, rendering conventional static models inadequate for encapsulating the developing environmental, regulatory, and behavioral elements. Recent advancements in machine learning (ML) and reinforcement learning (RL) provide adaptive, data-driven modeling for complex systems [1,2]. The Markov decision process is appropriate for sequential decision-making in uncertain conditions. When integrated with deep learning for prediction, MDPs provide optimum methods that conform to both economic objectives and environmental regulations [3,4]. This article presents a hybrid ML-MDP framework that amalgamates deep neural networks for prediction, the Markov decision process for decision modeling, and reinforcement learning techniques (e.g., value iteration, Q-learning) for policy optimization. We substantiate the methodology using a municipal solid waste management example, demonstrating enhanced performance relative to static and rule-based benchmarks across critical variables, including cost, emissions, and compliance.
2. Materials and Methods
2.1. System Architecture
The suggested structure consists of the following complementary modules.
- Predictive module: A deep neural network (DNN) anticipates future resource flow utilizing past data (e.g., trash creation, recycling, energy recovery) [5,6].
- Decision module: A finite-horizon Markov Decision Process (MDP) characterizes dynamic transitions, employing DNN outputs to assess actions like routing, incentives, and scheduling within a reward framework grounded in environmental and operational key performance indicators (KPIs) [7].
- Optimization module: Value Iteration and Q-Learning algorithms produce policies that maximize long-term rewards in intricate, uncertain situations.
2.2. Dataset and Preprocessing
We utilize an empirical municipal solid waste dataset from British Columbia (2017–2022), encompassing daily waste quantities, recycling metrics, energy recovery statistics, seasonal variables, and expenditure records. Missing values are subjected to linear interpolation, features undergo MinMax scaling, and the data is divided chronologically into training (70%), validation (15%), and test (15%) groups.
2.3. Neural Network Architecture
The prediction model is a multilayer perceptron (MLP) including 10 input features and three hidden layers of 128, 64, and 32 units, respectively, employing rectified linear unit activation. The output layer employs a linear function. Training reduces mean squared error (MSE) with the Adam optimizer with a learning rate of 0.001. Regularization encompasses dropout (0.2) and early stopping (patience = 20). The model undergoes training for 300 epochs with a batch size of 64, and cross-validation is utilized to mitigate overfitting. The loss function is defined as follows:
2.4. MDP Formulation
MDP is characterized as a tuple (S, A, P, R, γ), in which S denotes system states (waste, recovery, cost), A represents potential actions like rerouting or frequency adjustment, P signifies the transition probability acquired using a noise-injected neural network, R symbolizes the reward function grounded in environmental and economic KPIs, and γ is the discount factor for long-term optimization.
A reward function is used as Equation (2).
2.5. Policy Optimization and Evaluation
We use Q-learning to update the action-value function:
A pseudocode was created as follows:
Algorithm 1 outlines the training procedure of the proposed model-based deep Q-learning framework, where the agent leverages a learned state transition model to update the Q-function through iterative interaction and planning.
| Algorithm 1: Model-Based Deep Q-Learning |
| Require: State transition model fθ(s, a), reward function R(s, a), learning rate η, discount factor γ, exploration rate ε, episodes M, horizon T Ensure: Trained Q-function Q(s, a) 1: for episode = 1 to M do 2: Initialize state s 3: for t = 1 to T do 4: With probability ε, select a random action a 5: Otherwise, select a ← arg max_{a’} Q(s, a’) 6: Predict next state s’ ← fθ(s, a) 7: Observe reward r ← R(s, a) 8: Q(s, a) ← Q(s, a) + η [ r + γ max_{a’} Q(s’, a’) − Q(s, a) ] 9: s ← s’ 10: end for 11: end for |
The evaluation metrics used in this study include the long-term discounted reward () and the waste recovery rate (%).
The workflow of the developed ML–MDP circular economy decision support framework is presented in Figure 1.
Figure 1.
Workflow of ML–MDP circular economy decision support framework.
3. Results and Discussion
3.1. Predictive Accuracy
The Table 1 shown as neural forecasting model outperformed classical baselines, linear regression and autoregressive integrated moving average (ARIMA), based on root mean squared error (RMSE) [8] and mean absolute percentage error (MAPE) [9], achieving the highest accuracy across all metrics. The neural model achieved the lowest RMSE and MAPE, improving prediction accuracy by 21% over linear regression and 17% over ARIMA. The neural model achieved the lowest RMSE and MAPE, improving prediction accuracy by 21% over linear regression and 17% over ARIMA.
Table 1.
Forecasting performance across models.
3.2. Policy Simulation and Performance
Three planning strategies were assessed in this study: (1) rule-based scheduling (RBS), which depends on fixed logistics and static operational guidelines; (2) reactive policy (RP), which employs basic feedback mechanisms activated by predetermined thresholds; and (3) ML–MDP optimized policy, which utilizes a neural–MDP hybrid framework for adaptive and sequential decision-making in dynamic environments. The proposed ML–MDP model shown as Table 2 as outperformed all baselines in recovery efficiency and long-term rewards, converging significantly faster under dynamic simulation.
Table 2.
Circular economy policy comparison.
3.3. Robustness to Environmental Variability
The suggested ML–MDP framework surpassed conventional planning methods, with the best resource recovery rate (72.9%), maximum sustainability reward (524.1), minimal operational cost (USD 904/day), and quickest convergence (6 steps). The system rapidly adjusted within 3–5 steps to demand fluctuations, seasonal garbage surges, and logistical disruptions, sustaining a consistent recovery rate exceeding 65%.
3.4. Visual Summary
Figure 2 illustrates that the optimized policy continually surpasses the baseline, attaining more cumulative rewards across the episodes. The enhanced stability and efficacy of the proposed framework in managing dynamic circular economy systems are presented.
Figure 2.
Reward comparison across episodes for baseline and optimized policies.
Figure 3 displays the state-action visit frequency heatmap, demonstrating the extensive exploration coverage of the policy across several states (S1–S5) and actions (A1–A5). Increased visit frequency signifies favored or strategically important state-action pairs employed by the ML–MDP model, demonstrating strong exploration and balanced decision-making, essential for adaptive adaptation to dynamic and uncertain environments.
Figure 3.
State–action visit frequency showing policy exploration coverage.
4. Discussion
The proposed ML–MDP framework efficiently enhances circular economy operations, surpassing conventional rule-based and reactive approaches in recovery, cost, and sustainability measures [10]. By integrating predictive analytics with sequential decision-making, it effectively captures intricate material flow patterns and swiftly adjusts to disruptions using Q-learning. The results also demonstrate strong convergence and extensive policy exploration. The developed method relies on dependable historical data and precisely calibrated incentive settings. Future research is necessary to implement scalable techniques such as deep Q-Network or Actor-Critic to enhance performance at greater sizes.
5. Conclusions
In this research, we developed and tested an innovative decision support system that integrates neural networks and Markov decision processes for optimizing the circular economy. The proposed methodology for municipal solid waste management markedly enhanced forecasting precision (RMSE = 0.048, R2 = 0.904), attained superior resource recovery rates (+16.7% relative to rule-based scheduling), diminished operational expenses, and expedited policy alignment. Moreover, it demonstrated strong resilience to fluctuations in waste production and external disturbances. The method integrates predictive analytics with reward-driven planning, offering a versatile and scalable solution for several sustainability domains, such as energy circularity, water reuse, and industrial byproduct management. Subsequent research will improve model interpretability, integrate stakeholder participation, and expand reinforcement learning applications to wider urban environments.
Author Contributions
Y.M. designed the methodology, implemented the model, and drafted the manuscript. D.Q. supervised the study, validated results, and revised the manuscript. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
Dataset available on request from the authors.
Conflicts of Interest
The authors declare no conflicts of interest.
References
- Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.; Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton, A.; et al. Mastering the Game of Go without Human Knowledge. Nature 2017, 550, 354–359. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
- Puterman, M.L. Markov Decision Processes: Discrete Stochastic Dynamic Programming; Wiley-Interscience: New York, NY, USA, 1994. [Google Scholar]
- Vilaça, N.L.; Costa, M.G.F.; Costa Filho, C.F.F., Filho. A Hybrid Deep Neural Network Architecture for Day-Ahead Electricity Forecasting: Post-COVID Paradigm. Energies 2023, 16, 3546. [Google Scholar] [CrossRef] [Scilit]
- Li, J.; He, J.; Xu, Z. Sustainability and Material Flow Analysis of Wind Turbine Blade Recycling in China. Sustainability 2025, 17, 4307. [Google Scholar] [CrossRef] [Scilit]
- Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction, 2nd ed.; MIT Press: Cambridge, MA, USA, 2018. [Google Scholar]
- Kirchherr, J.; Reike, D.; Hekkert, M. Conceptualizing the circular economy: An analysis of 114 definitions. Resour. Conserv. Recycl. 2017, 127, 221–232. [Google Scholar] [CrossRef] [Scilit]
- Hyndman, R.J.; Koehler, A.B. Another look at measures of forecast accuracy. Int. J. Forecast. 2006, 22, 679–688. [Google Scholar] [CrossRef] [Scilit]
- Makridakis, S.; Wheelwright, S.C.; Hyndman, R.J. Forecasting: Methods and Applications, 3rd ed.; Wiley: New York, NY, USA, 1998. [Google Scholar]
- Ahamed, A.; Vallam, P.; Iyer, K.; Veksha, A.; Bobacka, J.; Lisak, G. Life cycle assessment of municipal solid waste management practices: A case study. J. Environ. Manag. 2019, 233, 767–776. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).


