Sensors

Research

Jump to: Review

30 pages, 43797 KB

Open AccessArticle

Modular Framework for Responsive and Explainable Robotic Assistance with Intention Prediction Using Human-Centric Digital Twins

by Usman Asad, Azfar Khalid, Waqas Akbar Lughmani, Shummaila Rasheed and Muhammad Mahabat Khan

Sensors 2026, 26(12), 3810; https://doi.org/10.3390/s26123810 - 15 Jun 2026

Viewed by 728

Abstract

Proactive robotic assistance in human–robot collaboration (HRC) requires systems that can perceive evolving task contexts, anticipate user needs, and intervene appropriately without disrupting human workflow. We present the Agentic Unified Robotic Assistance (AURA) Framework, which couples Large Language Model (LLM) reasoning grounded by [...] Read more.

Proactive robotic assistance in human–robot collaboration (HRC) requires systems that can perceive evolving task contexts, anticipate user needs, and intervene appropriately without disrupting human workflow. We present the Agentic Unified Robotic Assistance (AURA) Framework, which couples Large Language Model (LLM) reasoning grounded by Standard Operating Procedures (SOPs) with a modular layer of specialized Intent, Motion, Perception, Sound, Affordance, and Performance Monitors that supply structured context to a central decision-making module, making the framework reconfigurable and auditable without retraining or re-prompting. We introduce a human-in-the-loop teleoperation data collection methodology and an offline evaluation scheme with an Appropriateness Score (A-Score) tailored to proactive intervention timing, and release a benchmark dataset of annotated multimodal HRC episodes containing workspace and robot wrist camera videos, robot joint states, and labeled intervention events. Across three tasks of varying complexity, we observe progressive gains in intent prediction and decision-making as the modules are supplied with richer grounded context (prior-state memory and tracked object locations), with Combined F1 rising by over 20 points between context-poor and context-rich conditions. The structured grounding allows lightweight multimodal backbones such as Gemini 3.1 Flash Lite to perform on par with heavier reasoning-tier models at roughly one-fifth the inference latency. Together, these contributions establish a scalable framework, benchmark, and evaluation methodology for advancing proactive robotic assistance in collaborative environments. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

16 pages, 19013 KB

Open AccessArticle

Risk-Prioritized Experience Replay for Stable In-Hand Manipulation

by Yunsik Jung, Lingfeng Tao, Michael Bowman, Jiucai Zhang and Xiaoli Zhang

Sensors 2026, 26(12), 3633; https://doi.org/10.3390/s26123633 - 7 Jun 2026

Viewed by 483

Abstract

Deep reinforcement learning (DRL) has shown strong capability for multi-finger dexterous in-hand manipulation, where high-dimensional control and complex object interactions make policy learning challenging. However, many existing DRL approaches emphasize task completion and learning efficiency without explicitly accounting for manipulation risk, which can [...] Read more.

Deep reinforcement learning (DRL) has shown strong capability for multi-finger dexterous in-hand manipulation, where high-dimensional control and complex object interactions make policy learning challenging. However, many existing DRL approaches emphasize task completion and learning efficiency without explicitly accounting for manipulation risk, which can lead to overly aggressive behaviors and unstable object handling. This study proposes Risk-Prioritized Experience Replay (Risk-PER), a replay-sampling strategy that incorporates task-specific risk scores derived from prior transitions. The proposed method assigns each transition a risk score based on three binary indicators related to manipulation instability and then biases replay toward lower-risk experiences while still allowing the agent to learn from risk-related events. Risk-PER is integrated with Deep Deterministic Policy Gradient (DDPG) and evaluated in MuJoCo simulation on two Allegro Hand in-hand manipulation tasks involving a block and an egg. Across the evaluated settings, Risk-PER achieves higher success rates, lower manipulation risk, and more stable learning behavior than HER and reward–penalty-based risk-averse baselines. These results suggest that incorporating task-specific risk awareness into replay prioritization can improve both learning efficiency and manipulation stability in dexterous in-hand manipulation. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

35 pages, 27039 KB

Open AccessArticle

A Complete Grocery Pick-and-Pack Application Using a Computationally Lightweight Vision-Based Mobile Manipulator

by Thanavin Mansakul, Gilbert Tang, Phil Webb, Jamie Rice, Daniel Oakley and James Fowler

Sensors 2026, 26(9), 2860; https://doi.org/10.3390/s26092860 - 3 May 2026

Viewed by 1640

Abstract

Mobile manipulators have become essential platforms for autonomous tasks that demand high-quality performance and efficient operational processes. This paper presents a complete grocery pick-and-pack system for a mobile manipulator, integrating a graphical user interface (GUI) with an end-to-end vision-based grasp detection pipeline designed [...] Read more.

Mobile manipulators have become essential platforms for autonomous tasks that demand high-quality performance and efficient operational processes. This paper presents a complete grocery pick-and-pack system for a mobile manipulator, integrating a graphical user interface (GUI) with an end-to-end vision-based grasp detection pipeline designed for lightweight computation. The system is evaluated on the Grocery Pick-and-Pack Benchmark (Level-3), the most challenging level due to deformable objects, dimensional constraints, and strict grasp-point requirements. Experimental results demonstrate an average success rate of 92% across five item classes, with the deformable sweet bag the most challenging at 60% and an average execution time of 7.5 s on an edge device. The system achieves strong computational efficiency, reflected by a compute-to-speed ratio (CSR) of 0.008, with a total model size of only 30.9 MB. Performance is further validated across multiple hardware platforms and under real competition scenarios in the European Robotics League 2025. The findings highlight the practical impact of lightweight, vision-based mobile manipulation and provide insights into current challenges and future research directions for autonomous robotic applications. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

15 pages, 2219 KB

Open AccessArticle

One Patch Is All You Need: Joint Surface Material Reconstruction and Classification from Minimal Visual Cues

by Sindhuja Penchala, Gavin Money, Gabriel Marques, Samuel Wood, Jessica Kirschman, Travis Atkison, Shahram Rahimi and Noorbakhsh Amiri Golilarz

Sensors 2026, 26(7), 2083; https://doi.org/10.3390/s26072083 - 27 Mar 2026

Viewed by 696

Abstract

Understanding material surfaces from sparse visual cues is critical for applications in robotics, simulation and material perception. However, most existing methods rely on dense or full scene observations, limiting their effectiveness in constrained or partial view environments. This gap highlights the need for [...] Read more.

Understanding material surfaces from sparse visual cues is critical for applications in robotics, simulation and material perception. However, most existing methods rely on dense or full scene observations, limiting their effectiveness in constrained or partial view environments. This gap highlights the need for models capable of inferring surfaces’ properties from extremely limited visual information. To address this challenge, we introduce SMARC, a unified model for Surface MAterial Reconstruction and Classification from minimal visual input. By giving only a single 10% contiguous patch of the image, SMARC recognizes and reconstructs the full RGB surface while simultaneously classifying the material category. Our architecture combines a Partial Convolutional U-Net with a classification head, enabling both spatial inpainting and semantic understanding under extreme observation sparsity. We compared SMARC against five models including convolutional autoencoders, Vision Transformer (ViT), Masked Autoencoder (MAE), Swin Transformer and DETR using the Touch and Go dataset of real-world surface textures. SMARC achieves the highest performance among the evaluated methods with a PSNR of 17.55 dB and a surface classification accuracy of 85.10%. These results validate the effectiveness of SMARC in relation to surface material understanding and highlight its potential for deployment in robotic perception tasks where visual access is inherently limited. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

17 pages, 561 KB

Open AccessArticle

Multimodal Shared Autonomy for Heavy-Load UAV Operations with Physics-Aware Cooperative Control

by Xu Gao, Jingfeng Wu, Yuchen Wang, Can Cao, Lihui Wang, Bowen Wang and Yimeng Zhang

Sensors 2026, 26(6), 1997; https://doi.org/10.3390/s26061997 - 23 Mar 2026

Viewed by 812

Abstract

Heavy-load unmanned aerial vehicles (UAVs) are increasingly being applied in logistics, infrastructure installation, and emergency response missions, where complex payload dynamics and unstructured environments pose significant challenges to safe and efficient operation. Conventional manual teleoperation interfaces, such as dual-joystick control, impose a high [...] Read more.

Heavy-load unmanned aerial vehicles (UAVs) are increasingly being applied in logistics, infrastructure installation, and emergency response missions, where complex payload dynamics and unstructured environments pose significant challenges to safe and efficient operation. Conventional manual teleoperation interfaces, such as dual-joystick control, impose a high cognitive workload and provide limited support for expressing high-level operator intent, while fully autonomous solutions remain difficult to deploy reliably under real-world uncertainty. To address these limitations, this paper proposes the Multimodal Fusion Cooperation Network (MFCN), an end-to-end shared autonomy framework that integrates speech commands, visual gestures, and haptic cues through cross-modal feature fusion to infer operator intent in real time. The fused intent representation is translated into dynamically feasible control commands by a cooperative control policy with embedded physics-aware constraints to suppress payload oscillations and ensure flight stability. Extensive semi-physical simulations and real-world experiments demonstrate that the MFCN significantly improves the task success rate, positioning accuracy, and payload stability while reducing the task completion time and operator cognitive workload compared with manual, unimodal, and heuristic multimodal baselines. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

18 pages, 2996 KB

Open AccessArticle

A Multimodal Agentic AI Framework for Intuitive Human–Robot Collaboration

by Xiaoyun Liang and Jiannan Cai

Sensors 2026, 26(6), 1958; https://doi.org/10.3390/s26061958 - 20 Mar 2026

Viewed by 5227

Abstract

Widespread acceptance of collaborative robots in human-involved scenarios requires accessible and intuitive interfaces for lay workers and non-expert users. Existing interfaces often rely on users to plan and issue low-level commands, necessitating extensive knowledge of robot control. This study proposes a multimodal agentic [...] Read more.

Widespread acceptance of collaborative robots in human-involved scenarios requires accessible and intuitive interfaces for lay workers and non-expert users. Existing interfaces often rely on users to plan and issue low-level commands, necessitating extensive knowledge of robot control. This study proposes a multimodal agentic AI framework integrating natural user interfaces (NUIs) to foster effortless human-like partnerships in human–robot collaboration (HRC), which enhance intuitiveness and operational efficiency. First, it allows users to instruct robots using plain language verbally, coupled with gaze, revealing objects precisely. Second, it offloads users’ workload for robot motion planning by understanding context and reasoning task decomposition. Third, coordinating with AI agents built on large language models (LLMs), the system interprets users’ requests effectively and provides feedback to establish transparent communication. This proof-of-concept study included experiments to demonstrate a practical implementation of the agentic AI framework on a mobile manipulation robot in the collaborative task of human–robot wood assembly. Seven participants were recruited to interact with this AI-integrated agentic robotic system. Task performance and user experience metrics were measured in terms of completion time, intervention rate, NASA TLX survey for workload, and valuable insights of practical applications were summarized through a qualitative analysis. This study highlights the potential of NUIs and agentic AI-embodied robots to overcome existing HRC barriers and contributes to improving HRC intuitiveness and efficiency. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

16 pages, 3708 KB

Open AccessArticle

Myoelectric and Inertial Data Fusion Through a Novel Attention-Based Spatiotemporal Feature Extraction for Transhumeral Prosthetic Control: An Offline Analysis

by Andrea Tigrini, Alessandro Mengarelli, Ali H. Al-Timemy, Rami N. Khushaba, Rami Mobarak, Mara Scattolini, Gaith K. Sharba, Federica Verdini, Ennio Gambi and Laura Burattini

Sensors 2025, 25(18), 5920; https://doi.org/10.3390/s25185920 - 22 Sep 2025

Cited by 2 | Viewed by 1131

Abstract

This study proposes a feature extraction scheme that fuses accelerometric (ACC) and electromyographic (EMG) data to improve shoulder movement identification in individuals with transhumeral amputation, in whom the clinical need for intuitive control strategies enabling reliable activation of full-arm prostheses is underinvestigated. A [...] Read more.

This study proposes a feature extraction scheme that fuses accelerometric (ACC) and electromyographic (EMG) data to improve shoulder movement identification in individuals with transhumeral amputation, in whom the clinical need for intuitive control strategies enabling reliable activation of full-arm prostheses is underinvestigated. A novel spatiotemporal warping feature extraction architecture was employed to realize EMG and ACC information fusion at the feature level. EMG and ACC data were collected from six participants with intact limbs and four participants with transhumeral amputation using an NI USB-6009 device at 1000 Hz to support the proposed feature extraction scheme. For each participant, a leave-one-trial-out (LOTO) training and testing approach was used for developing pattern recognition models for both the intact-limb (IL) and amputee (AMP) groups. The analysis revealed that the introduction of ACC information has a positive impact when using windows of length (WLs) lower than 150 ms. A linear discriminant analysis (LDA) classifier was able to exceed the accuracy of 90% in each WL condition and for each group. Similar results were observed for an extreme learning machine (ELM), whereas k-nearest neighbors (kNN) and an autonomous learning multi-model classifier showed a mean accuracy of less than 87% for both IL and AMP groups at different WLs, guaranteeing applicability over a large set of shallow pattern-recognition models that can be used in real scenarios. The present work lays the groundwork for future studies involving real-time validation of the proposed methodology on a larger population, acknowledging the current limitation of offline analysis. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

16 pages, 5201 KB

Open AccessArticle

Robotic Fast Patch Clamp in Brain Slices Based on Stepwise Micropipette Navigation and Gigaseal Formation Control

by Jinyu Qiu, Qili Zhao, Ruimin Li, Yuzhu Liu, Biting Ma and Xin Zhao

Sensors 2025, 25(4), 1128; https://doi.org/10.3390/s25041128 - 13 Feb 2025

Cited by 5 | Viewed by 2299

Abstract

The patch clamp technique has become the gold standard for neuron electrophysiology research in brain science. Brain slices have been widely utilized as the targets of the patch clamp technique due to their higher optical transparency compared to a live brain and their [...] Read more.

The patch clamp technique has become the gold standard for neuron electrophysiology research in brain science. Brain slices have been widely utilized as the targets of the patch clamp technique due to their higher optical transparency compared to a live brain and their intercellular connectivity in comparison to cultured single neurons. However, the narrow working space, small scope, and depth of the field of view make the positioning of the operation’s micropipette to the target neuron a time-consuming task reliant on a high level of experience, significantly slowing down operation of the patch clamp technique in brain slices. Further, the current poor controllability in gigaseal formation, which is the key to electrophysiology signal recording, significantly lowers the patch clamp success rate. In this paper, a stepwise navigation of the micropipette is conducted to accelerate the positioning process of the micropipette tip to the target neuron in the brain slice. Then, a fuzzy proportional–integral–derivative controller is designed to control the gigaseal formation process along a designed resistance curve. The experimental results demonstrate an almost doubled patch clamp technique speed, with a 25% improvement in the success rate compared to the conventional manual method. The above advantages may promote the application of our method in brain science research based on brain slice platforms. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

23 pages, 4654 KB

Open AccessArticle

Effective Acoustic Model-Based Beamforming Training for Static and Dynamic Hri Applications

by Alejandro Luzanto, Nicolás Bohmer, Rodrigo Mahu, Eduardo Alvarado, Richard M. Stern and Néstor Becerra Yoma

Sensors 2024, 24(20), 6644; https://doi.org/10.3390/s24206644 - 15 Oct 2024

Cited by 3 | Viewed by 4569

Abstract

Human–robot collaboration will play an important role in the fourth industrial revolution in applications related to hostile environments, mining, industry, forestry, education, natural disaster and defense. Effective collaboration requires robots to understand human intentions and tasks, which involves advanced user profiling. Voice-based communication, [...] Read more.

Human–robot collaboration will play an important role in the fourth industrial revolution in applications related to hostile environments, mining, industry, forestry, education, natural disaster and defense. Effective collaboration requires robots to understand human intentions and tasks, which involves advanced user profiling. Voice-based communication, rich in complex information, is key to this. Beamforming, a technology that enhances speech signals, can help robots extract semantic, emotional, or health-related information from speech. This paper describes the implementation of a system that provides substantially improved signal-to-noise ratio (SNR) and speech recognition accuracy to a moving robotic platform for use in human–robot interaction (HRI) applications in static and dynamic contexts. This study focuses on training deep learning-based beamformers using acoustic model-based multi-style training with measured room impulse responses (RIRs). The results show that this approach outperforms training with simulated RIRs or matched measured RIRs, especially in dynamic conditions involving robot motion. The findings suggest that training with a broad range of measured RIRs is sufficient for effective HRI in various environments, making additional data recording or augmentation unnecessary. This research demonstrates that deep learning-based beamforming can significantly improve HRI performance, particularly in challenging acoustic environments, surpassing traditional beamforming methods. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

Review

Jump to: Research

32 pages, 2874 KB

Open AccessReview

Survey on Reconnaissance Autonomous Robotic Systems for Disaster Management

by Sahaj Sinha, Sinjae Lee and Saurabh Singh

Sensors 2026, 26(5), 1659; https://doi.org/10.3390/s26051659 - 5 Mar 2026

Cited by 2 | Viewed by 1655

Abstract

Systems that operate in dangerous environments are becoming essential in case of emergencies. This survey reviews the latest ground reconnaissance robots using computer vision (CV), machine learning (ML), MCU-based control, LoRa communication, DC motors, and dual-power systems. The analysis includes hardware and algorithms, [...] Read more.

Systems that operate in dangerous environments are becoming essential in case of emergencies. This survey reviews the latest ground reconnaissance robots using computer vision (CV), machine learning (ML), MCU-based control, LoRa communication, DC motors, and dual-power systems. The analysis includes hardware and algorithms, and their performance in the field and lab. There has been clear progress in navigation, sensor fusion, and situational awareness. The main challenges which remain include the use of energy and standardization of benchmarks. This survey focuses exclusively on Unmanned Ground Vehicles (UGVs) for disaster reconnaissance, examining recent advances in hardware, software, and autonomy. The survey highlights the improvements in navigation, sensor fusion, and intelligence, and identifies remaining challenges such as energy limitations, robustness in harsh conditions, and the lack of standardized benchmarks. The analysis synthesizes findings from over 190 recent studies (2020–2025) in ground-based disaster robotics, providing a comprehensive overview of current capabilities and research gaps. It encapsulates all issues with their remedy for future disaster-response systems. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

51 pages, 4751 KB

Open AccessReview

Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy

by Vinit Mehta, Charu Sharma and Karthick Thiyagarajan

Sensors 2025, 25(20), 6394; https://doi.org/10.3390/s25206394 - 16 Oct 2025

Cited by 10 | Viewed by 7356

Abstract

With the rapid advancement of artificial intelligence and robotics, the integration of Large Language Models (LLMs) with 3D vision is emerging as a transformative approach to enhancing robotic sensing technologies. This convergence enables machines to perceive, reason, and interact with complex environments through [...] Read more.

With the rapid advancement of artificial intelligence and robotics, the integration of Large Language Models (LLMs) with 3D vision is emerging as a transformative approach to enhancing robotic sensing technologies. This convergence enables machines to perceive, reason, and interact with complex environments through natural language and spatial understanding, bridging the gap between linguistic intelligence and spatial perception. This review provides a comprehensive analysis of state-of-the-art methodologies, applications, and challenges at the intersection of LLMs and 3D vision, with a focus on next-generation robotic sensing technologies. We first introduce the foundational principles of LLMs and 3D data representations, followed by an in-depth examination of 3D sensing technologies critical for robotics. The review then explores key advancements in scene understanding, text-to-3D generation, object grounding, and embodied agents, highlighting cutting-edge techniques such as zero-shot 3D segmentation, dynamic scene synthesis, and language-guided manipulation. Furthermore, we discuss multimodal LLMs that integrate 3D data with touch, auditory, and thermal inputs, enhancing environmental comprehension and robotic decision-making. To support future research, we catalog benchmark datasets and evaluation metrics tailored for 3D-language and vision tasks. Finally, we identify key challenges and future research directions, including adaptive model architectures, enhanced cross-modal alignment, and real-time processing capabilities, which pave the way for more intelligent, context-aware, and autonomous robotic sensing systems. Full article

(This article belongs to the Special Issue Advanced Sensors and AI Integration for Human–Robot Teaming)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Advanced Sensors and AI Integration for Human–Robot Teaming

Share This Special Issue

Editors

Special Issue Information

Keywords

Benefits of Publishing in a Special Issue

Published Papers (11 papers)

Research

Review

Further Information

Guidelines

MDPI Initiatives

Follow MDPI