Search Results (553)

In the domain of algorithmic decision-making, non-Markovian dynamics manifest as a significant impediment, especially for paradigms such as Reinforcement Learning (RL), thereby exerting far-reaching consequences on the advancement and effectiveness of the associated systems. Nevertheless, the existing benchmarks are deficient in comprehensively assessing the capacity of decision algorithms to handle non-Markovian dynamics. To address this deficiency, we have devised a generalized methodology grounded in category theory. Notably, we established the category of Markov Decision Processes (MDP) and the category of non-Markovian Decision Processes (NMDP), and proved the equivalence relationship between them. This theoretical foundation provides a novel perspective for understanding and addressing non-Markovian dynamics. We further introduced non-Markovianity into decision-making problem settings via the History Aggregator for State (HAS). With HAS, we can precisely control the state dependency structure of decision-making problems in the time series. Our analysis demonstrates the effectiveness of our method in representing a broad range of non-Markovian dynamics. This approach facilitates a more rigorous and flexible evaluation of decision algorithms by testing them in problem settings where non-Markovian dynamics are explicitly constructed. Full article

As distributed generators (DGs) and flexible adjustable loads (FALs) further penetrate distribution networks (DNs), to reduce regulation complexity compared with traditional centralized control frameworks, DGs and FALs in DNs should be packed in several clusters to enable their dispatch to become standard in the industry. To mitigate the negative influence of DGs’ and FALs’ spatiotemporal distribution and uncertain output characteristics on dispatch, this paper proposes an intelligent dynamic cluster partitioning strategy for DNs, from which the DN’s resources and loads can be intelligently aggregated, organized, and regulated in a dynamic and optimal way with relatively high implementation efficiency. An environmental model based on the Markov decision process (MDP) technique is first developed for DN cluster partitioning, in which a continuous state space, a discrete action space, and a dispatching performance-oriented reward are designed. Then, a novel random forest Q-learning network (RF-QN) is developed to implement dynamic cluster partitioning by interacting with the proposed environmental model, from which the generalization and robust capability to estimate the Q-function can be improved by taking advantage of combining deep learning and decision trees. Finally, a modified IEEE-33-node system is adopted to verify the effectiveness of the proposed intelligent dynamic cluster partitioning and regulation strategy; the results also indicate that the proposed RF-QN is superior to the traditional deep Q-learning (DQN) model in terms of renewable energy accommodation rate, training efficiency, and portioning and regulation performance. Full article

Dynamic objects present a major challenge for visual simultaneous localization and mapping (Visual SLAM), as feature measurements originating from moving regions can corrupt camera pose estimation and lead to inaccurate maps. In this paper, we propose a lightweight, semantic-free front-end enhancement for ORB-SLAM that detects and suppresses dynamic features using optical flow geometry. The key idea is to estimate a global motion direction point (MDP) from optical flow vectors and to classify feature points based on their angular consistency with the camera-induced motion field. Unlike magnitude-based flow filtering, the proposed strategy exploits the geometric consistency of optical flow with respect to a motion direction point, providing robustness not only to depth variation and camera speed changes but also to different camera motion patterns, including pure translation and pure rotation. The method is integrated into the ORB-SLAM front-end without modifying the back-end optimization or cost function. Experiments on public dynamic-scene datasets demonstrate that the proposed approach reduces absolute trajectory error by up to approximately 45% compared to baseline ORB-SLAM, while maintaining real-time performance on a CPU-only platform. These results indicate that reliable dynamic feature suppression can be achieved without semantic priors or deep learning models. Full article

To address the low trajectory tracking accuracy and limited robustness of conventional reinforcement learning algorithms under complex marine environments involving wind, wave, and current disturbances, this study proposes a proximal policy optimization (PPO) algorithm incorporating an intrinsic curiosity mechanism to solve the unmanned surface vehicle (USV) trajectory tracking control problem. The proposed approach is developed on the basis of a three-degree-of-freedom (3-DOF) USV model and formulated within a Markov decision process (MDP) framework, where a multidimensional state space and a continuous action space are defined, and a multi-objective composite reward function is designed. By incorporating a pure pursuit guidance algorithm, the complexity of engineering implementation is reduced. Furthermore, an improved PPO algorithm integrated with an intrinsic curiosity mechanism is adopted as the trajectory tracking controller, in which the exploration incentives provided by the intrinsic curiosity module (ICM) guide the agent to explore the state space efficiently and converge rapidly to an optimal control policy. The final experimental results indicate that, compared with the conventional PPO algorithm, the improved PPO–ICM controller achieves a reduction of 54.2% in average lateral error and 47.1% in average heading error under simple trajectory conditions. Under the complex trajectory condition, the average lateral error and average heading error are reduced by 91.8% and 41.9%, respectively. These results effectively demonstrate that the proposed PPO–ICM algorithm attains high tracking accuracy and strong generalization capability across different trajectory scenarios, and can provide a valuable reference for the application of intelligent control algorithms in the USV domain. Full article

This study investigated the systemic metabolic effects of feeding a Fusarium-contaminated diet to early-lactation Holstein cows, with or without a mycotoxin-deactivating product (MDP; Mycofix^® Plus, BIOMIN Holding GmbH, Tulln, Austria). Thirty cows were divided into three dietary groups: a mildly contaminated control (CTR), a moderately contaminated diet containing zearalenone and deoxynivalenol (MTX), and the same contaminated diet supplemented with MDP. Plasma collected at 56 days in milk was analyzed by untargeted ultra-high-performance liquid chromatography (UHPLC) coupled with high-resolution mass spectrometry (HRMS), and multivariate models identified discriminant metabolites and pathways. MTX-fed cows showed alterations in sphingolipid metabolism, including accumulation of ceramide (t18:0/16:0), lactosylceramide, and sphinganine 1-phosphate, consistent with ceramide synthase inhibition and lipid remodeling stress. Increases in estradiol, estrone, and cholesterol sulfate suggested endocrine disruption, while elevated 8-oxo-dGMP indicated oxidative DNA damage. MDP supplementation mitigated these alterations, reducing sphingolipid intermediates, modulating tryptophan and glycerophospholipid pathways, and lowering oxidative stress markers. Metabolites such as riboflavin, pipecolic acid, and N-acetylserotonin could be likely associated with an improved mitochondrial function and redox homeostasis, although future studies are required to confirm this hypothesis. Additionally, MDP-fed cows exhibited distinct shifts in pyrimidine and nucleotide metabolism. Overall, MDP effectively counteracted Fusarium-related metabolic disturbances, supporting its protective role in maintaining lipid balance, hormonal stability, oxidative control, and metabolic resilience. Full article

Background/Objectives: Mueller–Weiss syndrome (MWS) is a rare condition characterized by spontaneous adult-onset osteonecrosis of the navicular bone. This study aimed to assess the diagnostic value of quantitative and semi-quantitative standardized uptake value (SUV) measurements on Tc-99m MDP SPECT/CT for differentiating MWS from other foot pathologies. Methods: We retrospectively reviewed 21 MWS patients who underwent SPECT/CT and compared them with 10 feet from 5 non-MWS patients as controls. MWS severity was staged using the Maceira classification. Volumes of interest (VOIs) were defined in the lateral navicular and distal tibia. SUV_max values were measured for the navicular bone (N), tibial metaphysis (Tm), and diaphysis (Td). Uptake ratios (N/Tm and N/Td) were calculated for semi-quantitative comparison. Results: MWS patients showed significantly higher SUV_max in the navicular compared with controls (9.2 vs. 1.5, p < 0.001). Both N/Tm and N/Td ratios were also significantly elevated (p < 0.001). SUV_max and uptake ratios positively correlated with Maceira stage and visual navicular uptake intensity. Diagnostic thresholds of N SUV_max > 3.77 (AUC = 0.93), N/Tm > 1.139 (AUC = 0.95), and N/Td > 0.93 (AUC = 0.93) effectively distinguished MWS from non-MWS cases. Conclusions: Quantitative and semi-quantitative SUV analysis on SPECT/CT offers a reliable tool for diagnosing MWS and evaluating disease severity. Semi-quantitative ratios, by normalizing metabolic variability, provide a practical and reproducible alternative to absolute SUV measurements for early detection and treatment planning in MWS. Full article

Background/Objectives: Tuberculosis (TB) remains the leading cause of death from a single infectious agent worldwide. In line with the World Health Organization’s (WHO) goal to end TB by 2035, the rapid development and clinical implementation of new, effective vaccines is urgently needed. To support global TB control efforts, we developed a novel candidate subunit multistage vaccine. Methods: This vaccine incorporates multiple Mycobacterium tuberculosis antigens expressed during both dormant and active stages of infection, fused into a single recombinant protein (ESAT6-CFP10-Ag85A-Rv2660c-Rv1813c). The antigen was encapsulated in biodegradable poly(D,L-lactide-co-glycolide) (PLGA) nanoparticles along with the pattern recognition receptor (PRR) agonists monophosphoryl lipid A (MPLA) and muramyl dipeptide (MDP), which function as adjuvants. Results: Using a mixed intramuscular/nasal prime-boost regimen, the vaccine elicited a mixed Th1/Th17 cell-mediated immune response, as well as a robust humoral response characterized by sustained systemic IgG (lasting at least one year) and prominent local secretory IgA. The vaccine demonstrated protective efficacy as a prophylactic booster following BCG priming in both murine and guinea pig models and was also effective in a therapeutic setting in a murine infection model. Conclusions: The results of this study provide empirical evidence that multistage tuberculosis vaccines represent a promising strategy for achieving global TB control. Full article

Cloud-native microservice architectures offer scalability and resilience but introduce complex interdependencies and new attack surfaces, making them vulnerable to resource-exhaustion Distributed Denial-of-Service (DDoS) attacks. These attacks propagate along service call chains, closely mimic legitimate traffic, and evade traditional detection and mitigation techniques, resulting in cascading bottlenecks and degraded Quality of Service (QoS). Existing Moving Target Defense (MTD) approaches lack adaptive, cost-aware policy guidance and are often ineffective against spatiotemporally adaptive adversaries. To address these challenges, this paper proposes ScaleShield, an adaptive MTD framework powered by Deep Reinforcement Learning (DRL) that learns coordinated, attack-aware defense policies for microservices. ScaleShield formulates defense as a Markov Decision Process (MDP) over multi-dimensional discrete actions, leveraging a Multi-Dimensional Double Deep Q-Network (MD3QN) to optimize service availability and minimize operational overhead. Experimental results demonstrate that ScaleShield achieves near 100% defense success rates and reduces compromised nodes to zero within approximately 5 steps, significantly outperforming state-of-the-art baselines. It lowers service latency by up to 72% under dynamic attacks while maintaining over 94% resource efficiency, providing robust and cost-effective protection against resource-exhaustion DDoS attacks in cloud-native environments. Full article

In the context of dual-carbon goals, Park-Level Integrated Energy Systems (PIES) are pivotal for enhancing renewable energy integration and promoting clean, efficient energy use. However, the inherent non-linearity from multi-energy coupling and the high dimensionality of operational data present substantial challenges for conventional scheduling optimization methods. To overcome these obstacles, this paper introduces a novel multi-objective scheduling framework for PIES leveraging deep reinforcement learning. We innovatively formulate the scheduling task as a Markov Decision Process (MDP) and employ the Trust Region Policy Optimization (TRPO) algorithm, which is adept at handling continuous action spaces. The state and action spaces are meticulously designed according to system constraints and user demands. A comprehensive reward function is then established to concurrently pursue three objectives: minimum operating cost, minimum carbon emissions, and maximum exergy efficiency. Through comparative analyses against other AI-based algorithms, our results demonstrate that the proposed method significantly lowers operating costs and carbon footprint while enhancing overall exergy efficiency. This validates the model’s effectiveness and superiority in addressing the complex multi-objective scheduling challenges inherent in modern energy systems. Full article

The increasing penetration of renewable energy resources is central to global sustainability and decarbonisation goals, yet it introduces intermittency and voltage instability in modern distribution networks. Ensuring stable operation while maximising renewable utilisation is critical for achieving long-term energy sustainability, reduced carbon emissions, and efficient grid performance. This study proposes a sustainability-oriented, Reinforcement Learning (RL)-driven voltage control framework that enables reliable and energy-efficient operation of wind-integrated distribution systems. A Deep Q-Network (DQN) agent formulates voltage regulation as a Markov Decision Process (MDP) and autonomously learns optimal control policies for on-load tap changers (OLTCs) and capacitor banks under highly variable wind and load conditions. Using the IEEE 33-bus test system with realistic stochastic wind and ZIP-load models, the results show that the proposed controller maintains voltages within statutory limits, reduces total active power losses by up to 18%, and enhances the network’s capacity to host renewable energy. These improvements translate to increased energy efficiency, reduced technical losses, and greater operational resilience, key enablers of sustainable energy distribution. The findings demonstrate that intelligent RL-based frameworks offer a scalable and model-free tool for advancing sustainable, low-carbon, and resilient power systems. Full article

Anemia, characterized by reduced hemoglobin (Hb), remains a major health concern. Although iron and erythropoietin (EPO) therapies are effective, limitations in safety and accessibility have prompted interest in nutritional alternatives. Hydrolyzed milk-derived peptides (H-MDPs) contain bioactive sequences with diverse physiological effects, yet their role in erythropoiesis remains poorly defined. This study investigated the hematopoietic actions of H-MDP using zebrafish and mouse models. Adult zebrafish underwent phlebotomy-induced anemia and received oral H-MDP for 3 weeks. Hb levels, erythrocyte morphology, and expression of erythropoiesis- and iron-metabolism genes were assessed. In healthy mice, renal Epo expression, circulating EPO, and serum cytokines were measured after 2 weeks of H-MDP administration. H-MDP significantly accelerated Hb recovery in anemic zebrafish (4.6 ± 0.64 g/dL vs. 3.4 ± 0.66 g/dL in untreated fish at week 1) and markedly improved erythrocyte maturation. These effects coincided with strong induction of epo, hif1aa/b, igf1, csf1a, and csf3b in the heart and liver, as well as normalization of anemia-induced hepatic iron-transport genes (tfa, fpn1, tfr2) and reactivation of hamp. In mice, H-MDP elevated renal Epo mRNA and circulating EPO (approximately 2.3-fold) without altering steady-state Hb, and cytokine profiling with IPA-predicted activation of the erythropoietin signaling pathway. Collectively, these findings indicate that H-MDPs modulate erythropoiesis by coordinating the activation of EPO-related and iron-regulatory networks, supporting their potential as functional food ingredients for hematologic recovery and anemia management. Full article

To enhance the navigation accuracy and obstacle avoidance capability of Unmanned Aerial Vehicles (UAVs) operating in dynamic multi-obstacle environments, this paper proposes an intelligent navigation and avoidance strategy based on deep reinforcement learning. First, a performance index is formulated by integrating miss distances from multiple obstacles with energy consumption. An optimal avoidance strategy is then derived as an expert policy through a solution of the Riccati equation. Subsequently, a Markov Decision Process (MDP) model is constructed for UAV navigation and obstacle avoidance, incorporating a multi-objective reward function that simultaneously optimizes avoidance success rate, navigation accuracy, and energy efficiency. Furthermore, a hybrid learning architecture combining Generative Adversarial Imitation Learning (GAIL) with Proximal Policy Optimization (PPO) is designed and trained. Simulation results demonstrate that the proposed method achieves high training efficiency and enables robust decision-making in complex navigation scenarios. Full article

A condition-based maintenance decision-making framework for multi-component systems is proposed in this work by integrating dynamic Bayesian network (DBN) with proportional hazards model (PHM). The framework is designed to address the challenge of handling mixed failure types and complex failure dependencies, which often lead to inaccurate maintenance decisions in existing methods. In this integrated model, the DBN captures the failure evolution and both dynamic and static dependencies among components, while the PHM enhances the capability to characterize mixed failure interactions, thereby enabling the coverage of three common types of failure dependencies in multi-component systems. The model is formulated and solved using a finite-horizon Markov decision process (MDP), with the optimal maintenance strategy obtained by maximizing the total expected reward. Numerical case studies demonstrate the framework’s flexibility in handling mixed failures and complex dependencies, showing its potential to effectively support condition-based maintenance decision-making for complex multi-component systems. Full article

This manuscript introduces a generalized Markov Decision Process (MDP) model for dynamic capacity planning in the presence of stochastic time-nonhomogeneous demand, wherein system capacity may be flexibly increased or decreased throughout a finite planning horizon. The model includes investment, disinvestment, maintenance, operational, and shortage costs, in addition to a salvage value at the end of the planning horizon. Under very realistic conditions, we investigate the structural properties of the optimal policy and demonstrate its monotonic structure. By leveraging these properties, we propose a revised value iteration algorithm that capitalizes on the intrinsic structure of the problem, thereby achieving enhanced computational efficiency compared to traditional dynamic programming techniques. The proposed model is applicable across a range of sectors, including manufacturing systems, cloud-computing services, logistics systems, healthcare resource management, power capacity planning, and other intelligent infrastructures driven by Industry 4.0. Full article

This paper addresses the retail pricing problem for electricity retailers who also act as virtual power plant (VPP) operators, aggregating distributed energy resources (DERs). In future power markets where multiple such retailers compete for customers, a key challenge is to design pricing strategies that balance economic profitability with low-carbon objectives. Existing research often overlooks the impact of retailers’ heterogeneous resource portfolios, particularly the share of low-carbon resources like photovoltaics (PVs), on their competitive advantage and pricing decisions. To bridge this gap, we propose a novel retail pricing model that integrates a non-cooperative game framework with Markov Decision Processes (MDPs). The model enables each retailer to formulate optimal real-time pricing strategies by anticipating competitors’ actions and customer responses, ultimately reaching a Nash equilibrium. A distinctive feature of our approach is the incorporation of spatially differentiated carbon emission factors, which are adjusted based on each retailer’s share of PV generation. This creates a tangible low-carbon incentive, allowing retailers with greener resource mixes to leverage their environmental advantage. The proposed framework is validated on a modified IEEE 30-bus system with six competing retailers. Simulation results demonstrate that our method effectively incentivizes optimal load distribution, alleviates network congestion, and improves branch loading indices. Critically, retailers with a higher share of PV resources achieved significantly higher profits, directly translating their low-carbon advantage into economic value. Notably, the Branch Load Index (BLI) was reduced by 12% and node voltage deviations were improved by 1.32% at Bus 12, demonstrating the model’s effectiveness in integrating economic and low-carbon objectives. Full article