Multimodal Shared Autonomy for Heavy-Load UAV Operations with Physics-Aware Cooperative Control
Abstract
1. Introduction
- We formulate heavy-load UAV shared autonomy as a multimodal intent-conditioned control problem and propose the MFCN to infer operator intent from synchronized speech, gesture, and haptic streams through learned cross-modal temporal reasoning rather than rule-based modality arbitration.
- We introduce a physics-aware cooperative control policy that converts the inferred intent and the UAV–payload state into dynamically feasible actions while explicitly regularizing payload swing energy growth and unsafe behavior.
- We establish a multi-stage evaluation protocol spanning semi-physical simulation, HIL validation, public benchmark perception tests, and real flight experiments, showing consistent improvements in task success, positioning accuracy, swing suppression, and operator workload over manual, unimodal, heuristic multimodal, and autonomy-only baselines.
2. Related Work
2.1. Shared Autonomy for Human–UAV Cooperation
2.2. Heavy-Load UAV Control with Suspended Payloads
2.3. Multimodal Interaction and Fusion in Robotics
3. Methodology
3.1. Framework Overview
3.2. Multimodal Perception and Feature Encoding
3.3. Cross-Modal Fusion for Intent Inference
3.4. Cooperative Control Policy
3.5. Physics-Aware Safety Regularization
3.6. Training and Implementation Details
4. Experiments
4.1. Experimental Setup
4.2. Baselines and Control Architectures
4.3. Quantitative Results in Simulation
4.4. Module-Specific Evaluation on Public Datasets
4.5. Real-World Results and Cognitive Load
4.6. Ablation Study and Feature Importance
4.7. Robustness Under Perception Degradation
5. Discussion
5.1. Impact of Multimodal Fusion on Shared Autonomy
5.2. Role of Physics-Aware Regularization
5.3. Operator Workload and Trust
5.4. Robustness and Generalization
5.5. Limitations and Future Work
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
| UAV | Unmanned Aerial Vehicle |
| MFCN | Multimodal Fusion Cooperation Network |
| DRL | Deep Reinforcement Learning |
| MPC | Model Predictive Control |
| JT | Manual Teleoperation |
| AO | Autonomous Optimization |
| UM | Unimodal Shared Autonomy |
| HM | Heuristic Multimodal |
| HIL | Hardware-in-the-Loop |
| ROS | Robot Operating System |
| IMU | Inertial Measurement Unit |
| CNN | Convolutional Neural Network |
| LSTM | Long Short-Term Memory |
| PPO | Proximal Policy Optimization |
| NASA-TLX | NASA Task Load Index |
References
- Khamseh, H.B.; Janabi-Sharifi, F.; Abdessameud, A. Aerial manipulation—A literature survey. Robot. Auton. Syst. 2018, 107, 221–235. [Google Scholar] [CrossRef] [Scilit]
- Kotarski, D.; Piljek, P.; Pranjić, M.; Kasać, J. Concept of a modular multirotor heavy lift unmanned aerial vehicle platform. Aerospace 2023, 10, 528. [Google Scholar] [CrossRef] [Scilit]
- Sarghini, F.; De Vivo, A. Analysis of preliminary design requirements of a heavy lift multirotor drone for agricultural use. Chem. Eng. Trans. 2017, 58, 625–630. [Google Scholar]
- Grant, G.; Belfadel, D.; Haessig, D. Design and Testing of heavy-lift UAV for Obstacle Avoidance Using Multi-sensor Fusion. In Proceedings of the 2025 Northeast Section Conference, Erie, PA, USA, 27–30 March 2025. [Google Scholar]
- Peksa, J.; Mamchur, D. A review on the state of the art in copter drones and flight control systems. Sensors 2024, 24, 3349. [Google Scholar] [CrossRef] [Scilit]
- Singh, R.; Kumar, S. A Comprehensive Insights into Drones: History, Classification, Architecture, Navigation, Applications, Challenges, and Future Trends. arXiv 2025, arXiv:2501.10066. [Google Scholar] [CrossRef] [Scilit]
- Willis, P. A Lifting Job. Aerosp. Test. Int. 2024, 2024, 1. [Google Scholar] [CrossRef] [Scilit]
- Darshankumar, J.; Mathankumar, M.; Arunkumar, A. Design and Fabrication of Multirotor UAV for Medical Applications. In Proceedings of the 2025 3rd International Conference on Advancements in Electrical, Electronics, Communication, Computing and Automation (ICAECA); IEEE: New York, NY, USA, 2025; pp. 1–5. [Google Scholar]
- Imran; Li, J. UAV Design for Agricultural Applications. In UAV Aerodynamics and Crop Interaction: Revolutionizing Modern Agriculture with Drone; Springer: Berlin/Heidelberg, Germany, 2025; pp. 169–202. [Google Scholar]
- Kashi, M.; Ghadiri, H. Robust nonlinear control for quadrotor slung load system subject to external disturbances. In Proceedings of the 2024 10th International Conference on Artificial Intelligence and Robotics (QICAR); IEEE: New York, NY, USA, 2024; pp. 68–73. [Google Scholar]
- Abhinay, N.; Das, K.; Ghose, D. Robust Trajectory Tracking Control of a Multi-Rotor UAV Carrying a Cable Suspended Load. In Proceedings of the 2025 International Conference on Unmanned Aircraft Systems (ICUAS); IEEE: New York, NY, USA, 2025; pp. 416–423. [Google Scholar]
- Xing, S.; Xian, B.; Shao, P. Multi-Objective and Multi-State Nonlinear Model Predictive Control for Quadrotor Slung-Payload System. In Proceedings of the 2025 44th Chinese Control Conference (CCC); IEEE: New York, NY, USA, 2025; pp. 540–545. [Google Scholar]
- Telli, K.; Kraa, O.; Himeur, Y.; Ouamane, A.; Boumehraz, M.; Atalla, S.; Mansoor, W. A comprehensive review of recent research trends on unmanned aerial vehicles (uavs). Systems 2023, 11, 400. [Google Scholar] [CrossRef] [Scilit]
- Kang, H.; Li, H.; Zhang, J.; Lu, X.; Benes, B. Flycam: Multitouch gesture controlled drone gimbal photography. IEEE Robot. Autom. Lett. 2018, 3, 3717–3724. [Google Scholar] [CrossRef] [Scilit]
- Ng, K.W.; Mahony, R.; Lau, D. A dual joystick-trackball interface for accurate and time-efficient teleoperation of cable-driven parallel robots within large workspaces. In Proceedings of the International Conference on Cable-Driven Parallel Robots; Springer: Berlin/Heidelberg, Germany, 2019; pp. 391–402. [Google Scholar]
- Yakushiji, F.; Yakushiji, K.; Murata, M.; Hiroi, N.; Takeda, K.; Fujita, H. The quality of blood is not affected by drone transport: An evidential study of the unmanned aerial vehicle conveyance of transfusion material in Japan. Drones 2020, 4, 4. [Google Scholar] [CrossRef] [Scilit]
- Paas, A.; Coffey, E.B.; Beltrame, G.; St-Onge, D. Towards evaluating the impact of swarm robotic control strategy on operators’ cognitive load. In Proceedings of the 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN); IEEE: New York, NY, USA, 2022; pp. 217–223. [Google Scholar]
- Cardenas, J.J.G.; Hei, X.; Tapus, A. Exploring cognitive load dynamics in human-machine interaction for teleoperation: A user-centric perspective on remote operation system design. In Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); IEEE: New York, NY, USA, 2024; pp. 12204–12211. [Google Scholar]
- Chandarana, M.; Meszaros, E.L.; Trujillo, A.; Allen, B.D. ’Fly Like This’: Natural Language Interface for UAV Mission Planning. In Proceedings of the International Conference on Advances in Computer-Human Interactions, Nice, France, 19–23 May 2017. [Google Scholar]
- Zhang, D.; Yang, G.; Khurshid, R.P. Haptic teleoperation of uavs through control barrier functions. IEEE Trans. Haptics 2020, 13, 109–115. [Google Scholar] [CrossRef] [Scilit]
- Weishi, C.; Yifeng, H.; Xianfeng, L. Survey on application of multi-sensor fusion in UAV detection technology. Mod. Radar 2020, 42, 15–29. [Google Scholar]
- Dragan, A.D.; Srinivasa, S.S. A policy-blending formalism for shared control. Int. J. Robot. Res. 2013, 32, 790–805. [Google Scholar] [CrossRef] [Scilit]
- Baratta, A.; Cimino, A.; Gnoni, M.G.; Longo, F. Human robot collaboration in industry 4.0: A literature review. Procedia Comput. Sci. 2023, 217, 1887–1895. [Google Scholar] [CrossRef] [Scilit]
- Perera, A.G.; Wei Law, Y.; Chahl, J. UAV-GESTURE: A dataset for UAV control and gesture recognition. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Anand, S.S.; Mathiyazaghan, R. Design and fabrication of voice controlled unmanned aerial vehicle. IAES Int. J. Robot. Autom 2016, 5, 205–212. [Google Scholar] [CrossRef] [Scilit]
- Su, H.; Qi, W.; Chen, J.; Yang, C.; Sandoval, J.; Laribi, M.A. Recent advancements in multimodal human–robot interaction. Front. Neurorobot. 2023, 17, 1084000. [Google Scholar] [CrossRef] [Scilit]
- Wang, T.; Zheng, P.; Li, S.; Wang, L. Multimodal human–robot interaction for human-centric smart manufacturing: A survey. Adv. Intell. Syst. 2024, 6, 2300359. [Google Scholar] [CrossRef] [Scilit]
- Oviatt, S. Multimodal interfaces. In The Human-Computer Interaction Handbook; CRC Press: Boca Raton, FL, USA, 2007; pp. 439–458. [Google Scholar]
- Bolt, R.A. “Put-that-there” Voice and gesture at the graphics interface. In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, Seattle, WA, USA, 14–18 July 1980; pp. 262–270. [Google Scholar]
- Yongda, D.; Fang, L.; Huang, X. Research on multimodal human-robot interaction based on speech and gesture. Comput. Electr. Eng. 2018, 72, 443–454. [Google Scholar] [CrossRef] [Scilit]
- Patel, R.V.; Atashzar, S.F.; Tavakoli, M. Haptic feedback and force-based teleoperation in surgical robotics. Proc. IEEE 2022, 110, 1012–1027. [Google Scholar] [CrossRef] [Scilit]
- Li, Y.; Zhang, F. Trust-preserved human-robot shared autonomy enabled by bayesian relational event modeling. IEEE Robot. Autom. Lett. 2024, 9, 10716–10723. [Google Scholar] [CrossRef] [Scilit]
- Franchi, A.; Secchi, C.; Ryll, M.; Bulthoff, H.H.; Giordano, P.R. Shared control: Balancing autonomy and human assistance with a group of quadrotor UAVs. IEEE Robot. Autom. Mag. 2012, 19, 57–68. [Google Scholar] [CrossRef] [Scilit]
- Li, G.; Li, Q.; Yang, C.; Su, Y.; Yuan, Z.; Wu, X. The classification and new trends of shared control strategies in telerobotic systems: A survey. IEEE Trans. Haptics 2023, 16, 118–133. [Google Scholar] [CrossRef] [Scilit]
- Ding, F.; Sun, C.; He, S. Anti-swing control for quadrotor-slung load transportation system with underactuated state constraints. Sensors 2023, 23, 8995. [Google Scholar] [CrossRef] [Scilit]
- Sreenath, K.; Lee, T.; Kumar, V. Geometric control and differential flatness of a quadrotor UAV with a cable-suspended load. In Proceedings of the 52nd IEEE Conference on Decision and Control; IEEE: New York, NY, USA, 2013; pp. 2269–2274. [Google Scholar]
- Lan, X.; Gong, L.; Zheng, L.; Liu, S.; Xu, W. Anti-swing strategy of a quadrotor with suspended payload based on model predictive control. In Proceedings of the 2023 9th International Conference on Control Science and Systems Engineering (ICCSSE); IEEE: New York, NY, USA, 2023; pp. 20–25. [Google Scholar]
- Wang, J.; Chu, J.; Liu, X. Anti-Swing Control for Quadrotor with Slung Load Using Integral Backstepping Sliding Mode Controller and Extended State Observer. In Proceedings of the 2024 China Automation Congress (CAC); IEEE: New York, NY, USA, 2024; pp. 6006–6011. [Google Scholar]
- Omar, H.M.; Akram, R.; Mukras, S.M.; Mahvouz, A.A. Recent advances and challenges in controlling quadrotors with suspended loads. Alex. Eng. J. 2023, 63, 253–270. [Google Scholar] [CrossRef] [Scilit]
- Mohammadi, M.; Franchi, A.; Barcelli, D.; Prattichizzo, D. Cooperative aerial tele-manipulation with haptic feedback. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); IEEE: New York, NY, USA, 2016; pp. 5092–5098. [Google Scholar]
- Yin Yuen, C.; Ng, K.K.; Liu, Y. Dynamic Modeling of Cable-Suspended Payload Quadcopter: A Physics-Informed Deep Learning Approach. In Proceedings of the AIAA AVIATION FORUM and ASCEND 2025, Las Vegas, NV, USA, 21–25 July 2025; p. 3242. [Google Scholar]
- Jia, Z.; Yu, J.; Mei, Y.; Chen, Y.; Shen, Y.; Ai, X. Integral backstepping sliding mode control for quadrotor helicopter under external uncertain disturbances. Aerosp. Sci. Technol. 2017, 68, 299–307. [Google Scholar] [CrossRef] [Scilit]
- Wang, H.; Li, N.; Wang, Y.; Su, B. Backstepping Sliding Mode Trajectory Tracking via Extended State Observer for Quadrotors with Wind Disturbance. Int. J. Control. Autom. Syst. 2021, 19, 3273–3284. [Google Scholar] [CrossRef] [Scilit]
- Chen, X.; Fan, Y.; Wang, G.; Mu, D. Improved extended disturbance observer-based integral backstepping sliding mode control for quadrotor slung-load system. Aerosp. Sci. Technol. 2025, 160, 110042. [Google Scholar] [CrossRef] [Scilit]
- Borja-Jaimes, V.; García-Morales, J.; Escobar-Jiménez, R.F.; Guerrero-Ramírez, G.V.; Adam-Medina, M. A Backstepping Sliding Mode Control of a Quadrotor UAV Using a Super-Twisting Observer. Appl. Sci. 2025, 15, 10120. [Google Scholar] [CrossRef] [Scilit]
- Zhou, A.; Han, L.; Meng, Y. Multimodal control of uav based on gesture, eye movement and voice interaction. In Proceedings of the International Conference on Guidance, Navigation and Control; Springer: Berlin/Heidelberg, Germany, 2022; pp. 3765–3774. [Google Scholar]
- Ghaleb, E.; Burenko, I.; Rasenberg, M.; Pouw, W.; Toni, I.; Uhrig, P.; Wilson, A.; Holler, J.; Özyürek, A.; Fernández, R. Leveraging Speech for Gesture Detection in Multimodal Communication. arXiv 2024, arXiv:2404.14952. [Google Scholar] [CrossRef] [Scilit]
- Choi, S.H.; Kim, Z.C.; Buu, S.J. Speech-Guided Drone Control System Based on Large Language Model. In Proceedings of the 2025 International Conference on Electronics, Information, and Communication (ICEIC); IEEE: New York, NY, USA, 2025; pp. 1–4. [Google Scholar]
- Sanders, B. Gesture Based Control of Semi-Autonomous Vehicles. Master’s Thesis, Old Dominion University, Norfolk, VA, USA, 2020. [Google Scholar]
- Nahri, S.N.F.; Du, S.; Van Wyk, B.J. A review on haptic bilateral teleoperation systems. J. Intell. Robot. Syst. 2022, 104, 13. [Google Scholar] [CrossRef] [Scilit]
- Sheridan, T.B. Human–robot interaction: Status and challenges. Hum. Factors 2016, 58, 525–532. [Google Scholar] [CrossRef] [Scilit]
- Rautiainen, S.; Pantano, M.; Traganos, K.; Ahmadi, S.; Saenz, J.; Mohammed, W.M.; Martinez Lastra, J.L. Multimodal interface for human–robot collaboration. Machines 2022, 10, 957. [Google Scholar] [CrossRef] [Scilit]
- Hu, W.; Deng, Z.; Yang, Y.; Zhang, P.; Cao, K.; Chu, D.; Zhang, B.; Cao, D. Socially Game-Theoretic Lane-Change for Autonomous Heavy Vehicle Based on Asymmetric Driving Aggressiveness. IEEE Trans. Veh. Technol. 2025, 74, 17005–17018. [Google Scholar] [CrossRef] [Scilit]
- Deng, Z.; Hu, W.; Sun, C.; Chu, D.; Huang, T.; Li, W.; Yu, C.; Pirani, M.; Cao, D.; Khajepour, A. Eliminating Uncertainty of Driver’s Social Preferences for Lane Change Decision-Making in Realistic Simulation Environment. IEEE Trans. Intell. Transp. Syst. 2025, 26, 1583–1597. [Google Scholar] [CrossRef] [Scilit]
- Deng, Z.; Hu, W.; Huang, T.; Sun, C.; Zhong, J.; Khajepour, A. Social Predictive Intelligent Driver Model for Autonomous Driving Simulation. Automot. Innov. 2025, 8, 1–12. [Google Scholar] [CrossRef] [Scilit]
- Karumbunathan, L.S. Nvidia jetson agx orin series. In A Giant Leap Forward for Robotics and Edge AI Applications. Technical Brief; NVIDIA Corporation: Santa Clara, CA, USA, 2022. [Google Scholar]
- Shotton, J.; Fitzgibbon, A.; Cook, M.; Sharp, T.; Finocchio, M.; Moore, R.; Kipman, A.; Blake, A. Real-time human pose recognition in parts from single depth images. In Proceedings of the CVPR 2011; IEEE: New York, NY, USA, 2011; pp. 1297–1304. [Google Scholar]
- Chitta, S.; Marder-Eppstein, E.; Meeussen, W.; Pradeep, V.; Tsouroukdissian, A.R.; Bohren, J.; Coleman, D.; Magyar, B.; Raiola, G.; Lüdtke, M.; et al. ros_control: A generic and simple control framework for ROS. J. Open Source Softw. 2017, 2, 456. [Google Scholar] [CrossRef] [Scilit]
- Malacarne, F. PX4 Autopilot Customization for Non-Standard Gimbal and UWB Peripherals. Ph.D. Thesis, Politecnico di Torino, Torino, Italy, 2020. [Google Scholar]
- Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. [Google Scholar] [CrossRef] [Scilit]
- Tsai, Y.H.H.; Bai, S.; Liang, P.P.; Kolter, J.Z.; Morency, L.P.; Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the Conference; Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; Volume 2019, p. 6558. [Google Scholar]
- Raissi, M.; Perdikaris, P.; Karniadakis, G.E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 2019, 378, 686–707. [Google Scholar] [CrossRef] [Scilit]
- Patrona, F.; Mademlis, I.; Pitas, I. An overview of hand gesture languages for autonomous UAV handling. In Proceedings of the 2021 Aerial Robotic Systems Physically Interacting with the Environment (AIRPHARO), Biograd na Moru, Croatia, 4–5 October 2021; pp. 1–7. [Google Scholar]
- Chéron, G.; Laptev, I.; Schmid, C. P-cnn: Pose-based cnn features for action recognition. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 3218–3226. [Google Scholar]
- Yang, F.; Wu, Y.; Sakti, S.; Nakamura, S. Make skeleton-based action recognition model smaller, faster and better. In Proceedings of the 1st ACM International Conference on Multimedia in Asia, Beijing, China, 16–18 December 2019; pp. 1–6. [Google Scholar]
- Papaioannidis, C.; Makrygiannis, D.; Mademlis, I.; Pitas, I. Learning fast and robust gesture recognition. In Proceedings of the 2021 29th European Signal Processing Conference (EUSIPCO); IEEE: New York, NY, USA, 2021; pp. 761–765. [Google Scholar]
- Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv 2018, arXiv:1804.03209. [Google Scholar] [CrossRef] [Scilit]
- Panayotov, V.; Chen, G.; Povey, D.; Khudanpur, S. Librispeech: An asr corpus based on public domain audio books. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); IEEE: New York, NY, USA, 2015; pp. 5206–5210. [Google Scholar]
- Samragh, M.; Kundu, A.; Hu, T.Y.; Chadha, A.; Srivastava, A.; Cho, M.; Tuzel, O.; Naik, D. I see what you hear: A vision-inspired method to localize words. In Proceedings of the ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); IEEE: New York, NY, USA, 2023; pp. 1–5. [Google Scholar]
- Bittar, A.; Dixon, P.; Samragh, M.; Nishu, K.; Naik, D. Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder. In Proceedings of the ICASSP 2024–2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); IEEE: New York, NY, USA, 2024; pp. 10386–10390. [Google Scholar]





| Method | Wind (m/s) | SR (%) | PosErr (m) | MaxSwing (°) | TTC (s) | NASA-TLX |
|---|---|---|---|---|---|---|
| JT | 2–3 | 82.0 ± 3.5 | 0.36 ± 0.07 | 12.1 ± 1.8 | 33.7 ± 4.1 | 72.4 ± 5.1 |
| JT | 4–6 | 40.5 ± 7.9 | 0.54 ± 0.12 | 17.8 ± 2.5 | 41.3 ± 5.8 | 81.7 ± 6.3 |
| AO | 2–3 | 88.4 ± 3.1 | 0.29 ± 0.05 | 8.4 ± 1.3 | 30.9 ± 3.7 | 68.2 ± 4.9 |
| AO | 4–6 | 67.1 ± 5.2 | 0.38 ± 0.08 | 11.2 ± 1.9 | 35.8 ± 4.5 | 70.1 ± 5.5 |
| MFCN | 2–3 | 95.2 ± 2.1 | 0.17 ± 0.03 | 5.8 ± 0.9 | 25.6 ± 3.2 | 49.3 ± 3.9 |
| MFCN | 4–6 | 83.1 ± 3.4 | 0.21 ± 0.04 | 7.1 ± 1.1 | 29.8 ± 3.8 | 55.8 ± 4.5 |
| Method | SR (%) | TTC (s) | PosErr (m) | MaxSwing (°) | SwingDecay (s) | CtrlEffort |
|---|---|---|---|---|---|---|
| JT (Teleop) | 56.2 ± 4.1 | 38.5 ± 5.2 | 0.42 ± 0.08 | 14.7 ± 2.1 | 9.1 ± 1.4 | 1.00 ± 0.12 |
| AO (Autonomy) | 78.1 ± 2.5 | 34.7 ± 3.9 | 0.29 ± 0.05 | 8.9 ± 1.3 | 6.5 ± 0.9 | 0.88 ± 0.07 |
| UM (Gesture-Only) | 74.3 ± 3.1 | 36.2 ± 4.1 | 0.31 ± 0.06 | 9.8 ± 1.5 | 7.0 ± 1.1 | 0.91 ± 0.09 |
| HM (Heuristic MM) | 81.5 ± 2.8 | 33.9 ± 3.5 | 0.27 ± 0.04 | 8.2 ± 1.2 | 6.0 ± 0.8 | 0.87 ± 0.06 |
| MFCN (Ours) | 92.4 ± 1.9 | 27.6 ± 2.8 | 0.18 ± 0.03 | 6.2 ± 0.8 | 4.3 ± 0.6 | 0.79 ± 0.05 |
| Method | AUTH UAV Gesture (%) | UAV-Gesture (%) |
|---|---|---|
| P-CNN [64] | - | 91.9 |
| DD-Net [65] | 74.2 | 91.5 |
| MLP [66] | 76.2 | 94.8 |
| Ours | 82.6 | 96.3 |
| Dataset | Ours | BC-ResNet [69] | KWS [70] |
|---|---|---|---|
| Speech Commands v2 [67] | 0.97 | 0.95 | 0.96 |
| LibriSpeech [68] | 0.91 | 0.86 | 0.89 |
| Variant | SR (%) | MaxSwing (°) | PosErr (m) | IntentAcc (%) | Comment |
|---|---|---|---|---|---|
| Full MFCN | 92.4 | 6.2 | 0.18 | 94.1 | — |
| w/o Physics loss | 83.7 | 9.1 | 0.25 | 93.5 | Policy becomes less stable |
| w/o Haptic | 85.3 | 7.4 | 0.23 | 83.0 | Reduced urgency/interaction cue |
| w/o Speech | 88.0 | 6.9 | 0.20 | 88.5 | Weaker symbolic instruction channel |
| w/o Gesture | 86.7 | 7.3 | 0.22 | 86.1 | Reduced spatial grounding |
| Early Fusion (concat) | 87.4 | 7.2 | 0.21 | 89.2 | Limited temporal cross-modal modeling |
| Late Fusion (gating) | 88.2 | 6.8 | 0.20 | 91.0 | Less adaptive under partial occlusion |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.
Share and Cite
Gao, X.; Wu, J.; Wang, Y.; Cao, C.; Wang, L.; Wang, B.; Zhang, Y. Multimodal Shared Autonomy for Heavy-Load UAV Operations with Physics-Aware Cooperative Control. Sensors 2026, 26, 1997. https://doi.org/10.3390/s26061997
Gao X, Wu J, Wang Y, Cao C, Wang L, Wang B, Zhang Y. Multimodal Shared Autonomy for Heavy-Load UAV Operations with Physics-Aware Cooperative Control. Sensors. 2026; 26(6):1997. https://doi.org/10.3390/s26061997
Chicago/Turabian StyleGao, Xu, Jingfeng Wu, Yuchen Wang, Can Cao, Lihui Wang, Bowen Wang, and Yimeng Zhang. 2026. "Multimodal Shared Autonomy for Heavy-Load UAV Operations with Physics-Aware Cooperative Control" Sensors 26, no. 6: 1997. https://doi.org/10.3390/s26061997
APA StyleGao, X., Wu, J., Wang, Y., Cao, C., Wang, L., Wang, B., & Zhang, Y. (2026). Multimodal Shared Autonomy for Heavy-Load UAV Operations with Physics-Aware Cooperative Control. Sensors, 26(6), 1997. https://doi.org/10.3390/s26061997
