Search Results (585)

Collaborative robots, or cobots, are designed to improve productivity and safety in industrial settings. However, effective Human–Robot Collaboration (HRC) relies heavily on the human operator’s trust in the robotic partner. This study posits that trust is significantly enhanced by the robot’s ability to adapt to unpredictable human behavior. To achieve this adaptability, we propose applying an Adversarial Reinforcement Learning (ARL) framework to the robot’s activity planning. We model the assembly process as a Markov Decision Process (MDP) on a Directed Acyclic Graph (DAG). The robot learns an assembly policy using an on-policy algorithm while a simulated human agent, trained with the same algorithm, acts as an adversary that introduces disturbances and delays. We applied the proposed approach to a simple industrial case study and evaluated it on complex assembly sequences generated synthetically. Although the ARL-trained robot did not outperform conventional assembly optimization algorithms in terms of task completion time, it guaranteed robustness against human variability. This ensured task completion within a bounded timeframe regardless of human actions. By demonstrating consistent performance and adaptability in the face of uncertainty, the robot exhibits the Ability and Benevolence components of the ABI model of trust. This fosters a more resilient and trustworthy collaborative environment. Full article

Precise control over the physicochemical and biological properties of colloidal particles is essential for the rational design of functional soft materials. In this work, we report a simple and scalable strategy for generating modular dendron particles (MDPs) through the self-assembly of fully characterized small-molecule Bis-MPA dendrons that act as programmable molecular building blocks for colloidal particle formation. By systematically varying three structural domains—the inner functionality, methylene spacer length, and outer connector—we achieve tunable formation of MDPs ranging from nano- to microscale dimensions. Upon solvent evaporation under mild drying conditions, pre-assembled MDPs act as structure-directing seeds that guide the emergence of hierarchical surface morphologies with spiky, scaly, or spherical protrusions, depending on dendron architecture. Importantly, these assemblies exhibit good biocompatibility toward non-tumoral bronchial epithelial (NL-20) cells while displaying selective cytotoxicity toward Neuro-2a neuroblastoma cells, demonstrating that dendron molecular architecture alone can govern particle size, morphology, and biological response without external drug loading. Collectively, these findings highlight modular Bis-MPA dendrons as versatile building blocks for directing particle size, morphology, and biological response through controlled self-assembly and evaporation-driven structuring. Full article

The proliferation of Internet of Things (IoT) devices has introduced significant security challenges. Resource-constrained devices face sophisticated threats but lack the computational capacity for advanced security analysis. This study investigates optimal security task allocation in Cognitive IoT (CIoT) networks. It specifically examines when IoT devices should process security tasks locally or offload them to Mobile Edge Computing (MEC) servers. The problem is formulated as a Continuous-Time Markov Decision Process (CTMDP). The study demonstrates that the optimal offloading policy has a threshold structure. Security tasks are offloaded to MEC servers when the offloading queue length is below a critical threshold, $k^{\ast }$. Otherwise, tasks are processed locally. This structural property is robust to changes in MEC server configurations and threat arrival patterns. It ensures an optimal and easily implementable security policy under the exponential model. Theoretical analysis establishes upper bounds on the performance of AI-based security controllers using the same models. The results also show that standard model-free Q-learning algorithms can recover optimal thresholds without any prior knowledge of the system parameters. Simulations across multiple reinforcement learning architectures, including Q-learning, State–Action–Reward–State–Action (SARSA), and Deep Q-networks (DQN), confirm that all methods converge to the predicted threshold. This empirically validates the analytical findings. The threshold structure remains effective under practical imperfections such as imperfect sensing and parameter estimation errors. Systems maintain 85% to 93% of their optimal performance. This work extends threshold Markov Decision Process (MDP) analysis from classical queuing theory to the context of CIoT security offloading. It provides optimal and practical policies and model-free algorithms for use by resource-constrained devices. Full article

This paper proposes a Q-Learning-Based Distributed Energy-Efficient Routing (QDER) protocol for underwater acoustic sensor networks (UASNs). The routing problem is formulated as a Markov Decision Process (MDP) and a distributed Q-learning approach is proposed. Each sensor node is treated as an agent that independently selects its next-hop node based on a Q-table. The rewards function is designed that jointly considers node residual energy and depth information, enabling each node to learn an effective routing policy through distributed decision-making. Unlike centralized routing approaches that rely on extensive global information exchange, the proposed scheme allows nodes to make local decisions, thereby reducing communication overhead and energy consumption while maintaining efficient routing paths. In addition, link quality is designed in the reward to account for channel conditions, which improves the robustness of the routing strategy under noisy underwater acoustic environments. Simulation results demonstrate that the QDER achieves better system performance compared with Depth-Based Routing (DBR) and Deep Q-Network-Based Intelligent Routing (DQIR). Considering channel attenuation and noise, the proposed method with the link quality metric achieves improved network lifetime and energy efficiency. It also shows good robustness and adaptability under different signal-to-noise ratio (SNR) conditions. Full article

Low Earth orbit (LEO) satellite networks require real-time routing to cope with dynamic topology variations caused by continuous orbital motion. As an alternative to conventional routing approaches, deep reinforcement learning (DRL) has recently gained attention as an effective means for optimizing routing paths. To solve routing problems modeled as a grid-based Markov decision process (grid-based MDP), DRL methods such as CNN-based Dueling DQN have been proposed. However, these approaches are difficult to implement in practice. In particular, the substantial floating-point computation and memory traffic of CNN inference make real-time onboard inference challenging under the stringent power and resource constraints of satellite platforms. To address these constraints, this paper proposes an INT8 quantization and hardware–software co-design framework using heterogeneous SoC FPGA acceleration. We offload compute-intensive CNN inference to the programmable logic (PL), while the processing system (PS) orchestrates overall control and data movement, forming a collaborative PS–PL architecture. Furthermore, we integrate the NITI-style two-pass scaling with PS–PL exponent propagation to preserve end-to-end integer consistency without floating-point conversion. To demonstrate its practical onboard feasibility, we employ standard accelerator implementation choices—such as output-stationary scheduling and on-chip prefetching—and conduct an ablation study over independently tunable axes (PE array size and PS-side buffer reuse) to quantify their incremental contributions. Experimental results show that the proposed PS–PL cooperative scheme dramatically reduces computation time compared to a PS-only reference implementation on the same platform. Full article

The real-time economic dispatch of microgrids (MGs) is challenged by the high penetration of renewable energy and the resulting source–load uncertainties. Conventional optimization-based scheduling methods rely heavily on accurate probabilistic models and often suffer from high computational burdens, which limits their real-time applicability. To address these challenges, a data-driven deep reinforcement learning (DRL) framework is proposed for real-time microgrid energy management. The MG dispatch problem is formulated as a Markov decision process (MDP), and a Deep Deterministic Policy Gradient (DDPG) algorithm is adopted to efficiently handle the high-dimensional continuous action space of distributed generators and energy storage systems (ESS). The system state incorporates renewable generation, load demand, electricity price, and ESS operational conditions, while the reward function is designed as the negative of the operational cost with penalty terms for constraint violations. A continuous-action policy network is developed to directly generate control commands without action discretization, enabling smooth and flexible scheduling. Simulation studies are conducted on an extended European low-voltage microgrid test system under both deterministic and stochastic operating scenarios. The proposed approach is compared with model-based methods (MPC and MINLP) and representative DRL algorithms (SAC and PPO). The results show that the proposed DDPG-based strategy achieves competitive economic performance, fast convergence, and good adaptability to different initial ESS conditions. In stochastic environments, the proposed method maintains operating costs close to the optimal MINLP reference while significantly reducing the online computational time. These findings demonstrate that the proposed framework provides an efficient and practical solution for the real-time economic dispatch of microgrids with high renewable penetration. Full article

Real-time path planning for Unmanned Surface Vehicles (USVs) in complex marine environments remains challenging due to unstructured environments, ocean current disturbances, and dynamic obstacles. This paper proposes an improved Hybrid Safety and Reward-Sensitive Twin Delayed Deep Deterministic Policy Gradient (H_RS_TD3) algorithm and constructs a high-fidelity simulation environment based on GEBCO bathymetric data and CMEMS ocean current data. The path planning problem is formulated as a Markov Decision Process (MDP), where the state space incorporates multi-beam radar perception, ocean current disturbances, and relative goal information, while the action space outputs continuous thrust and rudder commands subject to vehicle dynamics constraints. The proposed framework integrates a risk-aware hybrid safety decision architecture, a Trajectory Predictor Network (TPN), a Curvature-driven Advantage-based Prioritized Experience Replay (CDA-PER) mechanism, and an uncertainty-aware conservative Q-learning strategy to enhance navigation safety, sample efficiency, and policy stability. Comprehensive simulations demonstrate that, compared with baseline deep reinforcement learning methods, the proposed approach achieves faster convergence, improved stability, and competitive path efficiency while consistently maintaining sufficient obstacle clearance and millisecond-level inference latency, validating its effectiveness and practical feasibility for safe USV navigation in realistic dynamic marine environments. Full article

Distributed Energy Resource (DER)-integrated power grids are vulnerable to cascading effects under concurrent cyber–physical attacks, where even minor disruptions in system states accumulate and amplify over time, leading to significant system failures. Traditional static risk assessment methods are insufficient for modeling these time-varying, dynamic scenarios, particularly in the context of concurrent attacks. This paper presents a dynamic risk assessment framework leveraging time-synchronized co-simulation, which integrates power system and communication network simulations within a unified time framework. Cyber-attack actions in the communication layer are mapped to corresponding physical disturbances in the distribution network, including voltage, frequency, and power variations. Using the resulting system state evolution trajectories, a Markov Decision Process (MDP)-based state transition tree captures the progression of system risk under concurrent attacks. This framework accounts for cumulative risk across different attack paths and identifies critical nodes and high-risk propagation paths within the network. By incorporating a concurrent event detector into the MDP model, the method quantifies evolving risk dynamics, overcoming the limitations of traditional static methods. Case studies on the IEEE 13-node test feeder and IEEE 14-bus system demonstrate that concurrent attacks result in a security risk metric 2.3 times higher than single-point attacks, validating the effectiveness of the proposed approach in identifying vulnerable nodes whose compromise could lead to cascading failures, supporting the risk-aware prioritization of defensive resources. Full article

Background: Post–core bonding plays a critical role in restoration longevity, and both the post surface treatment and core composite polymerization mode may influence interfacial performance. Methods: This in vitro study evaluated the effect of the post surface condition (no treatment vs. airborne-particle abrasion combined with an MDP-containing primer) and the composite polymerization mode (dual-, light-, and chemical-cure) on the pull-out bond strength and failure behavior of prefabricated metal post–core systems. A 3 × 2 factorial design was applied to 72 specimens (n = 12). After thermocycling, bond strength and failure modes were analyzed using two-way analysis of variance (ANOVA) and chi-square tests (p < 0.05). Results: The surface treatment significantly increased bond strength (p < 0.001; η² = 0.49) and shifted failure modes toward predominantly non-adhesive patterns (p = 0.011). Although the core type also showed a significant effect (p < 0.001), its influence was comparatively smaller. The bond strength was ranked as light-cure > chemical-cure > dual-cure under both surface conditions. Conclusions: Within the limitations of this study, post surface treatment was the primary determinant of bond strength and failure behavior. Clinically, effective surface modification appears to be more decisive than core composite selection, while differences among core materials become more apparent after establishing a stable bonding substrate. Full article

Navigation planning and execution for underwater gliders operating in thermocline-affected environments is challenging due to the coupled influence of energy constraints, spatially distributed environmental disturbances, and limited control authority. Spatially varying thermocline structures act as structured environmental disturbances that degrade motion efficiency and tracking accuracy, and therefore must be explicitly considered in both path planning and control design. This paper proposes a hierarchical control-oriented decision framework for underwater glider navigation in thermocline regions. At the planning layer, a thermocline-aware multi-objective optimization problem is formulated to regulate the trade-off between navigation efficiency and cumulative environmental disturbance, characterized by total path length and cumulative thermocline exposure, respectively. A multi-objective artificial bee colony (MOABC) algorithm is employed to generate a set of Pareto-optimal reference trajectories that explicitly reveal this trade-off. At the execution layer, pitch angle regulation is formulated as a stochastic tracking control problem under environmental uncertainty. A Markov Decision Process (MDP) is constructed to model the coupled effects of pitch control on energy consumption and trajectory deviation, and a deep deterministic policy gradient (DDPG) algorithm is adopted to synthesize a feedback control policy for adaptive pitch regulation during path execution. Simulation results demonstrate that the proposed framework effectively reduces cumulative thermocline exposure and overall energy consumption while maintaining improved trajectory consistency compared with representative benchmark methods. These results indicate that integrating multi-objective planning with learning-based control provides an effective control-oriented solution for constrained underwater glider navigation in thermally stratified environments. Full article

Unmanned aerial vehicles (UAVs) have emerged as a promising platform for supporting integrated sensing, computing, and communication (ISCC) functionality in Internet of Things (IoT) applications. This paper investigates a UAV-enabled ISCC network, where the UAV performs radar sensing and onboard edge computing with the computational assistance of ground access points (APs). Given the limited onboard energy, ensuring energy-efficient operation of UAVs is crucial to support the long-term sustainability of network performance. In this paper, we define computing efficiency as the ratio between the total number of successfully processed computational bits and the overall UAV energy consumption, under the constraint of a required sensing threshold. To maximize this performance metric, this paper jointly optimizes the beamforming vector, the CPU frequency, and the trajectory of the UAV. This optimization problem is modeled as a Markov decision process (MDP) and solved using a deep reinforcement learning (DRL) approach based on a memory mechanism. Specifically, a long short-term memory (LSTM) and twin delayed deep deterministic policy gradient (TD3)-based trajectory design and resource allocation (LTTDRA) algorithm is proposed. LSTM units are integrated into the actor and critic to effectively capture the temporal correlations in dynamic environments, thereby enhancing policy stability and accelerating algorithm convergence. The reward function is meticulously designed to alleviate sparse-penalty effects and learn high-performance strategies in complex environments with multiple constraints. Extensive simulations are conducted under various settings and network scenarios, and the results consistently indicate that the proposed approach substantially outperforms the baseline schemes. Full article

The optimization of the MOSFET gate drive parameters is crucial for the trade-off between switching loss and electromagnetic interference (EMI). However, the nonlinear coupling among gate drive parameters, board-level parasitic, and switching performance limits the effectiveness of traditional MOSFET drive design methods. This paper proposes an adaptive tuning framework based on the proximal policy optimization (PPO) algorithm. An analytical switching model incorporating board-level parasitics is first derived to analyze the coupling between drive parameters and switching performance. The optimization problem is then formulated as a Markov decision process (MDP). Within this framework, domain randomization is applied during training. This enables the agent to learn a generalizable optimization strategy that remains robust across the varying parasitic inductances encountered in different PCB layouts. Compared to the traditional Non-dominated Sorting Genetic Algorithm II (NSGA-II), the proposed method uses the trained policy for direct inference. This reduces computation time by $98.7\%$ while maintaining a multi-objective performance difference within $10.06\%$. In addition, hardware verification shows a 10.7% average deviation between the measured and simulated results. These results demonstrate that the proposed method provides an efficient and scalable solution for MOSFET gate drive optimization. Full article

Understanding the effects of mixing-driven precipitation on solute transport behavior is critical for reactive transport predictions, yet its complexity, arising from the interplay of flow dynamics, solute transport, and geochemical reactions, remains a significant challenge. In particular, mineral precipitation modifies the hydraulic properties of porous media. The impact of this process on the solute transport behavior remains largely unexplored and is crucial for accurate reactive transport predictions. This study presents a controlled laboratory investigation of mixing-driven calcite precipitation (MDP) in an intermediate-scale Hele-Shaw cell, simulating a coarse-sand porous medium. The experiment allowed for direct visualization of the spatiotemporal evolution of precipitation while continuously monitoring hydraulic properties. Self-organized heterogeneities in the precipitate structure were observed, with calcite layers forming symmetric patterns aligned with the main flow, contrasting with the asymmetry predicted by a semi-analytical model under idealized conditions. Tracer tests conducted before and after precipitation demonstrated significant impacts on solute transport, including the emergence of strong anomalous transport features, such as earlier solute arrival, a distinct double peak, and pronounced tailing. These findings highlight the critical role of precipitation-induced heterogeneities in shaping transport behavior, emphasizing the need to integrate these dynamics into reactive transport models for improved predictive accuracy. Full article

Unmanned aerial vehicle (UAV) guidance systems must operate reliably under significant uncertainties, such as sensor noise, target maneuvers, and environmental disturbances. Traditional guidance methods like proportional navigation (PN), while computationally efficient, often struggle to maintain performance under such challenging conditions. To overcome these limitations, this study proposes a robust UAV guidance framework based on deep reinforcement learning (DRL), specifically utilizing the soft actor–critic (SAC) algorithm. The UAV–target tracking problem is formulated as the Markov decision process (MDP) for both two-dimensional (2D) and three-dimensional (3D) scenarios. A deep neural network policy is trained in noisy environments to generate acceleration commands that minimize the zero-effort miss (ZEM). Extensive numerical simulations conducted using the OpenAI Gym validate effectiveness of the proposed method under previously unseen initial conditions and increased noise levels. The results demonstrate that the SAC-based policy achieves higher tracking success rates than the PN, particularly under strict terminal conditions and observation noise. Full article

As a next-generation immersive service, holographic video enables users to move freely within a virtual world. This imposes stringent requirements on wireless networks. Given the massive bandwidth capacity inherent to visible light, visible light communication (VLC) can effectively meet the transmission requirements of holographic video and is an ideal wireless technology for next-generation indoor immersive services. However, VLC channels are highly dependent on Line-of-Sight (LoS) links. Due to user mobility, traditional VLC systems relying on fixed-orientation Photodetectors (PDs) often suffer from severe channel fading, which significantly degrades the transmission performance. In this paper, we propose an indoor VLC holographic video transmission architecture supporting rotatable PDs, utilizing rotatable PDs mounted on Head-Mounted Displays (HMDs) to assist in holographic video transmission. To minimize the total transmission delay of all users, we address the holographic video transmission problem by jointly optimizing the transmit power allocation of VLC Access Points (APs) and the pitch and roll angles of the users’ PDs. By formulating the problem as a Markov Decision Process (MDP), we address it using a novel Deep Reinforcement Learning (DRL) strategy leveraging the Soft Actor–Critic (SAC) architecture. Simulation results demonstrate that the proposed scheme reduces the overall latency by up to 29.6% compared to the benchmark schemes. Furthermore, the convergence speed of the algorithm is improved by 35% compared to traditional deep reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG). Full article