Search Results (37)

This paper proposes a novel Twin Delayed Deep Deterministic Policy Gradient (TDDPG)-based joint optimization algorithm for hybrid reconfigurable intelligent surface (RIS)-assisted integrated sensing and communication (ISAC) systems in Internet of Vehicles (IoV) scenarios. The proposed system model achieves deep integration of sensing and communication by superimposing the communication and sensing signals within the same waveform. To decouple the complex joint design problem, a dual-DDPG architecture is introduced, in which one agent optimizes the transmit beamforming vector and the other adjusts the RIS phase shift matrix. Both agents share a unified reward function that comprehensively considers multi-user interference (MUI), total transmit power, RIS noise power, and sensing accuracy via the CRLB constraint. Simulation results demonstrate that the proposed TDDPG algorithm significantly outperforms conventional DDPG in terms of sum rate and interference suppression. Moreover, the adoption of a hybrid RIS enables an effective trade-off between communication performance and system energy efficiency, highlighting its practical deployment potential in dynamic IoV environments. Full article

We propose a tri-fold learning algorithm, called Neuroevolution of Hybrid Neural Networks in a Robotic Agent (acronym in Spanish, NRNH-AR), based on deep reinforcement learning (DRL), with self-supervised learning (SSL) and unsupervised learning (USL) steps, specifically designed to be implemented in a small autonomous navigation robot capable of operating in constrained physical environments. The NRNH-AR algorithm is designed for a small physical robotic agent with limited resources. The proposed algorithm was evaluated in four critical aspects: computational cost, learning stability, required memory size, and operation speed. The results obtained show that the performance of NRNH-AR is within the ranges of the Deep Q Network (DQN), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed Deep Deterministic Policy Gradient (TD3). The proposed algorithm comprises three types of learning algorithms: SSL, USL, and DRL. Thanks to the series of learning algorithms, the proposed algorithm optimizes the use of resources and demonstrates adaptability in dynamic environments, a crucial aspect of navigation robotics. By integrating computer vision techniques based on a Convolutional Neuronal Network (CNN), the algorithm enhances its abilities to understand visual observations of the environment rapidly and detect a specific object, avoiding obstacles. Full article

The organic Rankine cycle (ORC) is a key technology for the recovery of low-grade waste heat, but its efficient and stable operation is challenged by complex kinetic coupling. This paper proposes a model partitioning strategy based on gap measurement to construct a high-fidelity ORC system model and combines the setting of observer decoupling and multi-model switching strategies to reduce the coupling impact and enhance adaptability. For control optimization, the reinforcement learning method of deep deterministic Policy Gradient (DDPG) is adopted to break through the limitations of the traditional discrete action space and achieve precise optimization in the continuous space. The proposed DDPG-MGPC (Hybrid Model Predictive Control) framework significantly enhances robustness and adaptability through the synergy of reinforcement learning and model prediction. Simulation shows that, compared with the existing hybrid reinforcement learning and MPC methods, DDPG-MGPC has better tracking performance and anti-interference ability under dynamic working conditions, providing a more efficient solution for the practical application of ORC. Full article

Urban road traffic congestion poses significant challenges to sustainable mobility in smart cities. Traditional traffic light systems, reliant on static or semi-fixed timers, fail to adapt to dynamic traffic conditions, exacerbating congestion and limiting inclusivity. To address these limitations, this paper proposes H-ATLM (a hybrid adaptive traffic lights management), a system utilizing the deep deterministic policy gradient (DDPG) reinforcement learning algorithm to optimize traffic light timings dynamically based on real-time data. The system integrates advanced sensing technologies, such as cameras and inductive loops, to monitor traffic conditions and adaptively adjust signal phases. Experimental results demonstrate significant improvements, including reductions in congestion (up to 50%), increases in throughput (up to 149%), and decreases in clearance times (up to 84%). These findings open the door for integrating accessibility-focused features such as adaptive signaling for accessible vehicles, dedicated lanes for paratransit services, and prioritized traffic flows for inclusive mobility. Full article

Thermal processes with prolonged and variable delays pose considerable difficulties due to unpredictable system dynamics and external disturbances, often resulting in diminished control effectiveness. This work presents a hybrid control strategy that synthesizes deep reinforcement learning (DRL) strategies with nonlinear model predictive control (NMPC) to improve the robust control performance of a thermal process with a long time delay. In this approach, NMPC cost functions are formulated as learning functions to achieve control objectives in terms of thermal tracking and disturbance rejection, while an actor–critic (AC) reinforcement learning agent dynamically adjusts control actions through an adaptive policy based on the exploration and exploitation of real-time data about the thermal process. Unlike conventional NMPC approaches, the proposed framework removes the need for predefined terminal cost tuning and strict constraint formulations during the control execution at runtime, which are typically required to ensure robust stability. To assess performance, a comparative study was conducted evaluating NMPC against AC-based controllers built upon policy gradient algorithms such as the deep deterministic policy gradient (DDPG) and the twin delayed deep deterministic policy gradient (TD3). The proposed method was experimentally validated using a temperature control laboratory (TCLab) testbed featuring long and varying delays. Results demonstrate that while the NMPC–AC hybrid approach maintains tracking control performance comparable to NMPC, the proposed technique acquires adaptability while tracking and further strengthens robustness in the presence of uncertainties and disturbances under dynamic system conditions. These findings highlight the benefits of integrating DRL with NMPC to enhance reliability in thermal process control and optimize resource efficiency in thermal applications. Full article

Partial shading conditions reduce the efficiency of photovoltaic (PV) systems by introducing multiple local maxima in the power–voltage curve, complicating Maximum Power Point Tracking (MPPT). Traditional MPPT methods, such as Perturb and Observe (P&O) and Incremental Conductance (IC), frequently converge to local maxima, leading to suboptimal power extraction. This study proposes a hybrid reinforcement learning-based MPPT approach that combines fuzzy techniques with Deep Deterministic Policy Gradient (DDPG) to enhance tracking accuracy under partial shading. The method integrates fuzzy membership functions into the actor–critic structure, improving state representation and convergence speed. The proposed algorithm is evaluated in a simulated PV environment under various shading scenarios and benchmarked against conventional Perturb and Observe P&O and IC methods. Experimental results demonstrate that the Fuzzy–DDPG approach outperforms these classical techniques by achieving a higher tracking efficiency of 95%, compared to 85% for P&O and 88% for IC in average, while also minimizing steady-state oscillations. Additionally, the proposed method reduces tracking errors by up to 7.9% compared to conventional MPPT algorithms. These findings indicate that the combination of fuzzy logic and deep reinforcement learning provides a more adaptive and efficient MPPT solution, ensuring improved energy harvesting in dynamically changing conditions. Full article

Due to the limitations of pure electric power endurance, turbo-electric hybrid power systems, which offer a high power-to-weight ratio, present a reliable solution for medium- and large-sized vertical take-off and landing (VTOL) aircraft. Traditional energy management strategies often fail to minimize fuel consumption across the entire flight profile while meeting power demands under varying flight conditions. To address this issue, this paper proposes a deep reinforcement learning (DRL)-based energy management strategy (EMS) specifically designed for turbo-electric hybrid propulsion systems. Firstly, the proposed strategy employs a Prior Knowledge-Guided Deep Reinforcement Learning (PKGDRL) method, which integrates domain-specific knowledge into the Deep Deterministic Policy Gradient (DDPG) algorithm to improve learning efficiency and enhance fuel economy. Then, by narrowing the exploration space, the PKGDRL method accelerates convergence and achieves superior fuel and energy efficiency. Simulation results show that PKGDRL has a strong generalization capability in all operating conditions, with a fuel economy difference of only 1.6% from the offline benchmark of the optimization algorithm, and in addition, the PKG module enables the DRL method to achieve a huge improvement in terms of fuel economy and convergence rate. In particular, the prospect theory (PT) in the PKG module improves fuel economy by 0.81%. Future research will explore the application of PKGDRL in the direction of real-time total power prediction and adaptive energy management under complex operating conditions to enhance the generalization capability of EMS. Full article

Hybrid ships offer significant advantages in energy efficiency and environmental sustainability. However, their complex structures present challenges in developing effective energy management strategies to ensure optimal power distribution and stable, efficient operation of the power system. This study establishes a mathematical model of a hybrid system for a specific ship and proposes an energy management strategy based on the deep deterministic policy gradient (DDPG) algorithm, a reinforcement learning technique. The proposed strategy’s feasibility and effectiveness are validated through comparisons with alternative energy management strategies and real-world ship data. Simulation results demonstrate that the DDPG-based strategy optimizes the diesel engine’s operating conditions and reduces total fuel consumption by 3.6% compared to a strategy based on the deep Q-network (DQN) algorithm. Full article

The cooperative patrol control of multiple unmanned surface vehicles (Multi-USVs) in dynamic aquatic environments presents significant challenges in global coverage efficiency and system robustness. The study proposes a cooperative patrol control algorithm for multiple unmanned surface vehicles (Multi-USVs) based on a hybrid embedded task state information model and reward reshaping techniques, addressing global coverage challenges in dynamic aquatic environments. By integrating patrol, collaboration, and obstacle information graphs, the algorithm generates kinematically feasible control actions in real time and optimizes the exploration-cooperation trade-off through a dense reward structure. Simulation results demonstrate that the algorithm achieves 99.75% coverage in a 1 km × 1 km task area, reducing completion time by 23% and 74% compared to anti-flocking and partition scanning algorithms, respectively, while maintaining collision rates between agents (CRBAA) and obstacles (CRBAO) below 0.15% and 0.5%. Compared to DDPG, SAC, and PPO frameworks, the proposed training framework (TFMUSV) achieves 28% higher rewards with 40% smaller fluctuations in later training stages. This study provides an efficient and reliable solution for autonomous monitoring and search-rescue missions in complex aquatic environments. Full article

Owing to the global concern regarding fossil energy consumption and carbon emissions, the power supply for traditional diesel-driven ships is being replaced by low-carbon power sources, which include hydrogen energy generation and photovoltaic (PV) power generation. However, the uncertainty of shipboard PV power generation due to weather changes and ship motion variations has become an essential factor restricting the energy management of all-electric ships. In this paper, a deep reinforcement learning-based optimization algorithm is proposed for a green ship energy management system (EMS) coupled with hydrogen fuel cells (HFCs), lithium batteries, PV generation, an electric power propulsion system, and service loads. The focus of this study is reducing the total operation cost and improving energy efficiency by jointly optimizing power generation and voyage scheduling, considering shipboard PV uncertainty. To verify the effectiveness of the proposed method, real-world data for a hybrid hydrogen- and PV-driven ship are selected for conducting case studies under various sailing conditions. The numerical results demonstrate that, compared to those obtained with the Double DQN algorithm, the PPO algorithm, and the DDPG algorithm without considering the PV system, the proposed DDPG algorithm reduces the total economic cost by 1.36%, 0.96%, and 4.42%, while effectively allocating power between the hydrogen fuel cell and the lithium battery and considering the uncertainty of on-board PV generation. The proposed approach can reduce energy waste and enhance economic benefits, sustainability, and green energy utilization while satisfying the energy demand for all-electric ships. Full article

With the continuous progress of UAV technology and the rapid development of mobile edge computing (MEC), the UAV-assisted MEC system has shown great application potential in special fields such as disaster rescue and emergency response. However, traditional deep reinforcement learning (DRL) decision-making methods suffer from limitations such as difficulty in balancing multiple objectives and training convergence when making mixed action space decisions for UAV path planning and task offloading. This article innovatively proposes a hybrid decision framework based on the improved Dynamic Adaptive Genetic Optimization Algorithm (DAGOA) and soft actor–critic with hierarchical action decomposition, an uncertainty-quantified critic ensemble, and adaptive entropy temperature, where DAGOA performs an effective search and optimization in discrete action space, while SAC can perform fine control and adjustment in continuous action space. By combining the above algorithms, the joint optimization of drone path planning and task offloading can be achieved, improving the overall performance of the system. The experimental results show that the framework offers significant advantages in improving system performance, reducing energy consumption, and enhancing task completion efficiency. When the system adopts a hybrid decision framework, the reward score increases by a maximum of 153.53% compared to pure deep reinforcement learning algorithms for decision-making. Moreover, it can achieve an average improvement of 61.09% on the basis of various reinforcement learning algorithms such as proposed SAC, proximal policy optimization (PPO), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3). Full article

This paper proposes a novel reconfigurable intelligent surface (RIS)-assisted downlink hybrid free-space optics (FSO)/radio frequency (RF) space–air–ground integrated network (SAGIN) architecture, where the high altitude platform (HAP) converts the optical signal sent by the satellite into an electrical signal through optoelectronic conversion. The drone equipped with RIS dynamically adjusts the signal path to serve ground users, thereby addressing communication challenges caused by RF link blockages from clouds or buildings. To improve the security performance of SAGIN, this paper maximizes the sum secrecy rate (SSR) by optimizing the power allocation, RIS phase shift, and drone trajectory. Then, an alternating iterative framework is proposed for a joint solution using the simulated annealing algorithm, semi-definite programming, and the designed deep deterministic policy gradient (DDPG) algorithm. The simulation results show that the proposed scheme can significantly enhance security performance. Specifically, compared with the NOMA and SDMA schemes, the SSR of the proposed scheme is increased by 39.7% and 286.7%, respectively. Full article

An efficient energy management strategy (EMS) is crucial for the energy-saving and emission-reduction effects of electric vehicles. Research on deep reinforcement learning (DRL)-driven energy management systems (EMSs) has made significant strides in the global automotive industry. However, most scholars study only the impact of a single DRL algorithm on EMS performance, ignoring the potential improvement in optimization objectives that different DRL algorithms can offer under the same benchmark. This paper focuses on the control strategy of hybrid energy storage systems (HESSs) comprising lithium-ion batteries and ultracapacitors. Firstly, an equivalent model of the HESS is established based on dynamic experiments. Secondly, a regulated decision-making framework is constructed by uniformly setting the action space, state space, reward function, and hyperparameters of the agent for different DRL algorithms. To compare the control performances of the HESS under various EMSs, the regulation properties are analyzed with the standard driving cycle condition. Finally, the simulation results indicate that the EMS powered by a deep Q network (DQN) markedly diminishes the detrimental impact of peak current on the battery. Furthermore, the EMS based on a deep deterministic policy gradient (DDPG) reduces energy loss by 28.3%, and the economic efficiency of the EMS based on dynamic programming (DP) is improved to 0.7%. Full article

Energy scheduling for hybrid unmanned aerial vehicles (UAVs) is of critical importance to their safe and stable operation. However, traditional approaches, predominantly rule-based, often lack the dynamic adaptability and stability necessary to address the complexities of changing operational environments. To overcome these limitations, this paper proposes a novel energy scheduling framework that integrates the Model Predictive Control (MPC) with a Deep Reinforcement Learning algorithm, specifically the Deep Deterministic Policy Gradient (DDPG). The proposed method is designed to optimize energy management in hydrogen-powered UAVs across diverse flight missions. The energy system comprises a proton exchange membrane fuel cell (PEMFC), a lithium-ion battery, and a hydrogen storage tank, enabling robust optimization through the synergistic application of MPC and DDPG. The simulation results demonstrate that the MPC effectively minimizes electric power consumption under various flight conditions, while the DDPG achieves convergence and facilitates efficient scheduling. By leveraging advanced mechanisms, including continuous action space representation, efficient policy learning, experience replay, and target networks, the proposed approach significantly enhances optimization performance and system stability in complex, continuous decision-making scenarios. Full article

Due to the uncertainty and complexity of the assembly process, the trajectory planning of a robot needs to consider the real-time obstacle avoidance problem when it completes the assembly in the unstructured workspace. To realize the safe assembly of assembly robots in dynamic and complex environments, a dynamic obstacle avoidance trajectory planning method for robots combining traditional planning algorithms and deep reinforcement learning algorithms is proposed to improve the robot’s agent and obstacle avoidance ability in dynamic and complex environments. The Bidirectional Rapidly-exploring Random Tree (Bi-RRT) method is utilized as a global planner to plan the global optimal path quickly; considering the real-time nature of the assembly process, the Soft Actor-Critic (SAC) is used as a local obstacle avoider to avoid obstacles more accurately and to find the nearest node generated by the Bi-RRT during the planning process, which is regarded as the goal during the local obstacle avoidance to reduce the model’s complexity. By training and testing in the simulation engine and comparing with SAC, DDPG and DQN algorithms, the method can avoid obstacles in dynamic and complex environments more efficiently, which verifies that the proposed hybrid method can accomplish the high-precision planning task with a high success rate. Full article