Search Results (40)

This paper proposes AutoRL, a high-fidelity Robot Operating System 2 (ROS2)–Gazebo simulation pipeline that addresses a critical reproducibility gap in learning-based flight control: existing reinforcement learning (RL) frameworks for unmanned aerial vehicles (UAVs) lack deterministic, step-level coupling between control actions and physics updates. AutoRL enforces a strict one-to-one correspondence between agent actions and physics updates via blocking ROS2 service calls, preserving the Markov property required for stable policy learning and enabling verifiable reproducibility independent of the learning algorithm. A composite reward function jointly optimizes attitude tracking accuracy, oscillation suppression, actuator smoothness, and disturbance robustness. Its modular, service-oriented architecture provides a reusable framework for offline-trained RL research. A proximal policy optimization (PPO) controller trained within AutoRL validates the framework, demonstrating consistent convergence and stable performance across multiple independently seeded runs. Determinism was experimentally verified across two regimes: with Gaussian IMU noise disabled, repeated rollouts produced bit-identical trajectories, while with noise enabled, the measured distribution of trajectory divergence agreed with a reference distribution drawn from the declared sensor model, together confirming that the blocking service call architecture eliminates all non-stochastic sources of nondeterminism between the agent and the physics engine. The trained model is exported in a lightweight form compatible with embedded flight control firmware and remains adaptable across airframe configurations by automatically recomputing the control allocation matrix from configuration files. The total policy network contains only $\mathrm{10,628}$ trainable parameters, and inference was measured on a Cortex-M7 microcontroller at $598.7$ µs per step, $15.0\%$ of the 4 ms control period. Full article

Electricity theft remains a critical threat to power distribution infrastructure globally, with annual losses exceeding USD 89 billion and non-technical loss rates reaching 40% in developing economies. While machine learning has emerged as the dominant analytical approach for automated theft detection in smart grid environments, the field lacks a unifying framework that connects algorithm selection to the operational realities of Distribution System Operators (DSOs). Existing reviews catalogue methods and report benchmark metrics without addressing how detection paradigm selection should be aligned with data maturity, regulatory requirements, computational constraints, and institutional capacity. This review addresses that gap by systematically analysing 90 peer-reviewed studies published between 2015 and 2025, identified through structured multi-database searches, screened against explicit eligibility criteria, and graded with a formal five-criterion quality rubric, through a unified adversarial time-series formulation that provides a consistent analytical lens across all major learning paradigms. The analysis covers supervised ensemble methods, unsupervised and semi-supervised anomaly detection, deep learning architectures, including convolutional neural networks, long short-term memory networks and Transformer models, graph neural networks, federated learning, and explainable artificial intelligence. Key findings reveal that no single paradigm achieves optimality across all deployment dimensions simultaneously, that gradient boosting methods deliver near state-of-the-art performance with significantly lower computational overhead than deep learning, and that hybrid architectures achieve AUC-ROC scores of 0.95 to 0.98 on benchmark datasets but require complementary governance mechanisms to satisfy regulatory defensibility requirements. A lifecycle-aligned deployment framework and a layered detection architecture are proposed, offering practitioners a structured pathway from early AMI rollout through to advanced smart grid deployment. The principal outcomes of the review are a formal characterisation of which component of the detection problem each learning paradigm estimates, quality-graded and harmonised benchmark performance ranges, and a quantified illustrative analysis indicating that the proposed layered architecture can improve inspection productivity by roughly an order of magnitude at a fixed field budget. Four priority research challenges are identified: real-time edge detection, continual learning, multi-modal data fusion, and standardised benchmarking. Full article

The Traveling Salesman Problem (TSP) and the Vehicle Routing Problem (VRP) are two classical combinatorial optimization problems. In recent years, their drone-assisted variants, the Traveling Salesman Problem with Drones (TSP-D) and the Vehicle Routing Problem with Drones (VRP-D) have attracted growing attention. Generally, these problems are solved using exact algorithms or metaheuristic algorithms. However, as the problem complexity increases and the scale of instances grows, these approaches often become less efficient. In this paper, we propose a reinforcement learning method with a shared attention encoder and a hierarchical dual-decoder architecture, where truck–drone coordination is achieved by first decoding the truck’s next node and then conditionally decoding the drone action. To further explore the solution space of large-scale instances, the proposed method adopts a multi-rollout learning strategy. We conducted experiments on large-scale TSP-D and VRP-D instances, and the results show that this model outperforms traditional metaheuristic algorithms in terms of both solution quality and computational efficiency. Full article

Knowledge-intensive projects in research and development (R&D), software, and high-technology sectors are delivered by knowledge workers who each command several skills. When many such projects compete for a shared pool of multi-skilled workers, and neither project arrival times nor durations are known in advance, deciding who works on what at each moment becomes a sequential decision problem. Therefore, we propose the multi-skilled, multi-project dynamic scheduling problem with random project arrivals and uncertain durations and formulate it as a Markov decision process (MDP) that minimizes the total human resource cost. Since the cost-to-go function is computationally intractable, we develop a rollout-based approximate dynamic programming (ADP) algorithm that approximates it via Monte Carlo simulation embedded with a randomized base policy and restricts the action space to ten representative allocation policies. On benchmarks extended from the Project Scheduling Problem Library (PSPLIB), the proposed policy lowers the average total cost by 3.1% to 16.7% relative to Q-learning on medium- and large-scale instances while completing more projects with shorter delays; on small-scale instances Q-learning attains a lower nominal cost, revealing a trade-off among cost, completion rate, and delay. Full article

This study considers a 1-m-1 hybrid flow shop scheduling problem that simultaneously incorporates four practical constraints: lot streaming, no-wait, blocking, and sequence-dependent setup times. Although each of these characteristics has been studied individually in the literature, their joint consideration in a single HFS model has received little attention. The problem is motivated by a real-world order sequencing problem in insulation board manufacturing, where all four constraints arise simultaneously from the production process. To formally characterize the problem, we develop a mixed-integer programming formulation that captures all operational constraints. For practical-scale problems, we propose several dispatching heuristics that can obtain sufficiently good solutions within a short computation time. We further develop a genetic algorithm as an independent solution approach to obtain high-quality solutions close to the optimum within a reasonable computation time. Computational experiments on instances generated based on real insulation board production characteristics demonstrate that the proposed algorithms outperform a benchmark greedy rule, and sensitivity analyses reveal the effects of setup time magnitude and the number of parallel machines on scheduling performance. Full article

Decarbonising municipal solid waste (MSW) is a defining algorithmic challenge for smart cities: waste-to-energy (WtE), composting and material-recovery investments must be committed years ahead under deep uncertainty about household source-separation uptake and the governing policy instruments (landfill taxes, carbon prices, compost subsidies). We present a two-stage stochastic mixed-integer linear programming (MILP) framework, applied to the Attica region of Greece (Athens; 5122 t/day MSW) and calibrated to confirmed 2024 weighbridge data. The contribution is an integration strategy rather than a new technique: endogenous WtE capacity sizing (Special Ordered Sets of Type 2 (SOS2) piecewise-linear cost, economies-of-scale exponent 0.85), the bilinear capacity–build coupling linearised exactly by McCormick envelopes (one factor being binary), and Pigouvian shadow-price recovery of the optimal policy instruments are combined in a single-shot, gap-bounded MILP and embedded in a 10,000-run Latin-hypercube Monte Carlo layer over 12 parameters with Spearman sensitivity indices. The individual components are established; their joint formulation is, to our knowledge, new. The pipeline solves 10,415 MILP instances. Three results are policy-relevant: investment is robust to rollout uncertainty (VSS ≈ €0; EVPI ≈ €5.4 M, 0.15%); the carbon price alone explains ~80% of cost variance (ρ = +0.891); and, under the model’s calibration, the implied Pigouvian-optimal landfill tax (€1100–3300/t) indicates a binding landfill cap is needed to secure diversion. The framework transfers to any metropolitan MSW system facing decarbonisation and circular-economy mandates. Full article

Full-duplex (FD) technology holds great promise for enhancing the spectral efficiency of Mobile Ad Hoc Networks (MANETs) and Wireless Sensor Networks (WSNs). However, the practical performance gain of FD over Half-Duplex (HD) is highly sensitive to the dynamic nature of traffic loads and residual self-interference. Existing Optimal Dynamic Selection Strategies (ODSS) often rely on static workload assumptions within a single time window, failing to capture long-term traffic fluctuations. Consequently, applying instantaneous switching strategies in highly bursty environments necessitates excessively frequent mode switching (e.g., the switching frequency can approach the total number of time windows), incurring prohibitive signaling overhead and unignorable MAC-layer adaptation delays. To overcome these concrete bottlenecks, this paper proposes a comprehensive traffic-aware adaptive duplex mode selection framework. First, we model the multi-scale dynamic workload using Dynamic Activated Probability in Short-term (DAPS) and Long-term (DAPL), effectively characterizing both bursty traffic (via Beta distribution) and Markov-modulated stable traffic. Second, by integrating physical layer performance analysis, we define the Break-even Workload Point (BWP) to partition traffic into Oversaturated (OZ) and Unsaturated (UZ) Workload Zones (WZs). Furthermore, to handle unknown future traffic with low complexity, we propose the Pre-scheduling Duplex selection based on the Workload zone Estimation (PDWE) algorithm. PDWE leverages a Hidden Markov Model (HMM) combined with a Rollout algorithm to estimate hidden traffic states and adaptively pre-schedule duplex modes. Simulation results demonstrate that the proposed strategy achieves near-optimal throughput (approximately 91% of the ideal ODSS) while reducing the duplex switching frequency by two orders of magnitude compared to instantaneous switching strategies. This approach offers a robust cross-layer solution for next-generation self-organizing networks. Full article

Background: Stepped-wedge cluster randomized trials (SW-CRTs) are increasingly used to evaluate population-level interventions, but trial validity depends on timely cluster transitions. Rollout feasibility is often assumed rather than modelled. In the context of a planned national trial of CPR training kiosks, we developed a Monte Carlo simulation algorithm to quantify logistical feasibility under uncertainty. Methods: A stochastic Monte Carlo algorithm was implemented to simulate deploying 100 CPR kiosks across eight Canadian cities under four team structures. Inputs included productivity (0.8–1.2 kiosks/day), disruption probabilities (weather, venue access, technical failure, staff illness, transport delays), and cost parameters (salaries, per diems, travel). Each scenario was simulated across 3000 iterations. Outputs included per-city feasibility (p ≤ 60 days), total project duration, and risk–cost trade-offs. Results: Single-team strategies required 9–10 months for full rollout, with winter-exposed cities such as Halifax and Charlottetown having up to 30% probability of exceeding 60 days. Two-team strategies halved rollout time (4–5 months) and achieved >95% on-time rollout across cities. Adding a third onsite staff member reduced risk by 5–15% with modest additional cost (~CAD 1500–2000 per city). Risk–cost analysis identified two teams with three staff as the most reliable strategy. Conclusions: Monte Carlo simulation provides a practical framework for assessing rollout feasibility in SW-CRTs. Applied to CPR kiosk deployment, it highlights the importance of staffing, seasonality, and city-level context. The approach is generalizable to other national interventions requiring phased rollout under uncertainty. Full article

Multi-robot ensembles comprising several manipulators are commonly used in industrial settings to process non-deterministic flows of items loaded by an upstream source onto a shared transportation system. After the execution of a given task, the robots regularly deposit the items on a common output flow, which conveys the semi-finished material towards the downstream portion of the plant for further processing. The productivity and reliability of the entire process, which is affected by the plant layout, by the quality of the adopted scheduling and task assignment algorithms, and by the proper balancing of the input and output flows, may be degraded by random disturbances and transient conditions of the input flow. In this paper, a highly accurate event-based simulator of this kind of system is used in conjunction with a rollout algorithm to optimize the performance of the plant in all operating scenarios. The proposed method relies on a simulation of the plant that comprehensively considers the dynamic performance of the manipulators, their actual motion planning algorithms, the adopted scheduling and task assignment methods, and the regulation of the material flows. The simulation environment is built upon computationally efficient maps able to predict the execution time of the tasks assigned to the robots, considering all the determining factors, and on a representation of the manipulators themselves as finite state automata. The proposed formalization of the line balancing problem as a Markov Decision Process and the resulting rollout optimization method are shown to substantially improve the performance of the plant, even in challenging situations, and to be well suited to real-time implementation even on commodity hardware. Full article

This study presents a novel cyber-resilient, data-driven optimisation framework for real-time energy management in electric vehicle (EV)-integrated smart grids. The proposed framework integrates a hybrid optimisation engine—combining genetic algorithms and reinforcement learning—with a real-time analytics module to enable adaptive scheduling under uncertainty. It accounts for dynamic electricity pricing, EV mobility patterns, and grid load fluctuations, dynamically reallocating charging demand in response to evolving grid conditions. Unlike existing GA/RL schedulers, this framework uniquely integrates adaptive optimisation with resilient forecasting under incomplete data and lightweight blockchain-inspired cyber-defence, thereby addressing efficiency, accuracy, and security simultaneously. To ensure secure and trustworthy EV–grid communication, a lightweight blockchain-inspired protocol is incorporated, supported by an intrusion detection system (IDS) for cyber-attack mitigation. Empirical evaluation using European smart grid datasets demonstrates a daily peak demand reduction of 9.6% (from 33 kWh to 29.8 kWh), with a 27% decrease in energy delivered at the original peak hour and a redistribution of demand that increases delivery at 19:00 h by nearly 25%. Station utilisation became more balanced, with weekly peak normalised utilisation falling from 1.0 to 0.7. The forecasting module achieved a mean absolute error (MAE) of 0.25 kWh and a mean absolute percentage error (MAPE) below 20% even with up to 25% missing data. Among tested models, CatBoost outperformed LightGBM and XGBoost with an RMSE of 0.853 kWh and $R^2$ of 0.416. The IDS achieved 94.1% accuracy, an AUC of 0.97, and detected attacks within 50–300 ms, maintaining over 74% detection accuracy under 50% novel attack scenarios. The optimisation runtime remained below 0.4 s even at five times the nominal dataset scale. Additionally, the study outlines a conceptual extension to support location-based planning of charging infrastructure. This proposes the alignment of infrastructure roll-out with forecasted demand to enhance spatial deployment efficiency. While not implemented in the current framework, this forward-looking integration highlights opportunities for synchronising infrastructure development with dynamic usage patterns. Collectively, the findings confirm that the proposed approach is technically robust, operationally feasible, and adaptable to the evolving demands of intelligent EV–smart grid systems. Full article

With the increasing adoption of cooperative multi-UAV systems in applications such as cargo delivery and ground reconnaissance, the demand for scalable and efficient path planning methods has grown substantially. However, traditional heuristic algorithms are frequently trapped in local optima, require task-specific manual tuning, and exhibit limited generalization capabilities. Furthermore, their dependence on iterative optimization renders them unsuitable for large-scale real-time applications. To address these challenges, this paper introduces an end-to-end deep reinforcement learning framework that bypasses the reliance on handcrafted heuristic rules. The proposed method leverages an encoder–decoder architecture with multi-head attention (MHA), where the encoder generates embeddings for UAVs and task parameters, while the decoder dynamically selects actions based on contextual embeddings and enforces feasibility through a masking mechanism. The MHA module effectively models global spatial-task dependencies among nodes, enhancing solution quality. Additionally, we integrate a Multi-Start Greedy Rollout Baseline to evaluate diverse trajectories via parallelized greedy searches, thereby reducing policy gradient variance and improving training stability. Experiments demonstrated significant improvements in scalability, particularly in 100-node scenarios, where our method drastically reduced inference time compared to conventional methods, while maintaining a competitive path cost efficiency. A further validation on simulated mission environments and real-world geospatial data (sourced from Google Earth) underscored the robust generalization of the framework. This work advances large-scale UAV mission planning by offering a scalable, adaptive, and computationally efficient solution. Full article

This scoping review examines the evolving landscape of European Union (EU) legislation, as it pertains to the implementation of artificial intelligence (AI) in smart grid systems. By outlining the current regulatory landscape, including the General Data Protection Regulation (GDPR), the EU Artificial Intelligence Act, the EU Data Act, the EU Data Governance Act, the ePrivacy framework, the Network and Information Systems (NIS2) Directive, the EU Cyber Resilience Act, the EU Network Code on Cybersecurity for the electricity sector, and the EU Cybersecurity Act, it highlights both constraints and opportunities for stakeholders, including energy utilities, technology providers, and end-users. The analysis delves into regulatory barriers such as data protection requirements, algorithmic transparency mandates, and liability concerns that can limit the scope and scale of AI deployment. Technological challenges are also addressed, ranging from the integration of distributed energy resources and real-time data processing to cybersecurity and standardization issues. Despite these challenges, this review emphasizes how compliance with EU laws may ultimately boost consumer trust, promote ethical AI usage, and streamline the roll-out of robust, scalable smart grid solutions. The paper further explores stakeholder benefits, including enhanced grid stability, cost reductions through automation, and improved sustainability targets aligned with the EU’s broader energy and climate strategies. By synthesizing these findings, the review offers insights into policy gaps, technological enablers, and collaborative frameworks critical for accelerating AI-driven innovation in the energy sector, helping stakeholders navigate a complex regulatory environment while reaping its potential rewards. Full article

This study addresses the resource-constrained project scheduling problem with flexible structures and uncertain activity durations. The problem is formulated as a Markov decision process, with the optimal policy determined through stochastic dynamic programming. To mitigate the curse of dimensionality in large-scale problems, several approximate methods are proposed to derive suboptimal policies. In addition to traditional methods based on priority rules and metaheuristic algorithms, we focus on the application of rollout algorithms. To improve the computational efficiency of the rollout algorithms, only the best-performing priority rules are employed for action evaluation, and the common random numbers technique is also incorporated. Experimental results demonstrate that rollout algorithms significantly outperform priority rules and metaheuristics. The common random numbers technique not only enhances computational efficiency but also improves the accuracy of action selection. The post-rollout algorithm reduces computation time by 44.37% compared to the one-step rollout, with only a 0.02% performance gap. In addition, rollout algorithms perform more stably than other methods under different problem characteristics. Full article

The use of Reinforcement Learning (RL) techniques has been widely addressed in the literature to cope with capacity sharing in 5G Radio Access Network (RAN) slicing. These algorithms consider a training process to learn an optimal capacity sharing decision-making policy, which is later applied to the RAN environment during the inference stage. When relevant changes occur in the RAN, such as the deployment of new cells in the network, RL-based capacity sharing solutions require a re-training process to update the optimal decision-making policy, which may require long training times. To accelerate this process, this paper proposes a novel Transfer Learning (TL) approach for RL-based capacity sharing solutions in multi-cell scenarios that is implementable following the Open-RAN (O-RAN) architecture and exploits the availability of computing resources at the edge for conducting the training/inference processes. The proposed approach allows transferring the weights of the previously learned policy to learn the new policy to be used after the addition of new cells. The performance assessment of the TL solution highlights its capability to reduce the training process duration of the policies when adding new cells. Considering that the roll-out of 5G networks will continue for several years, TL can contribute to enhancing the practicality and feasibility of applying RL-based solutions for capacity sharing. Full article

Replacing mechanical utility meters with digital ones is crucial due to the numerous benefits they offer, including increased time resolution in measuring consumption, remote monitoring capabilities for operational efficiency, real-time data for informed decision-making, support for time-of-use billing, and integration with smart grids, leading to enhanced customer service, reduced energy waste, and progress towards environmental sustainability goals. However, the cost associated with replacing mechanical meters with their digital counterparts is a key factor contributing to the relatively slow roll-out of such devices. In this paper, we present a low-cost and power-efficient solution for retrofitting the existing metering infrastructure, based on state-of-the-art communication and artificial intelligence technologies. The edge device we developed contains a camera for capturing images of a dial meter, a 32-bit microcontroller capable of running the digit recognition algorithm, and an NB-IoT module with (E)GPRS fallback, which enables nearly ubiquitous connectivity even in difficult radio conditions. Our digit recognition methodology, based on the on-device training and inference, augmented with federated learning, achieves a high level of accuracy (97.01%) while minimizing the energy consumption and associated communication overhead (87 $\mathsf{\mu }$Wh per day on average). Full article