VR-Based Teleoperation of UAV–Manipulator Systems: From Single-UAV Control to Dual-UAV Cooperative Manipulation

Abstract

In this paper, we present a VR-based control framework for multi-UAV (rotorcraft-type) aerial manipulation that enables simultaneous control of each UAV and its onboard five-degree-of-freedom (5-DoF) manipulator using virtual-reality controllers. Instead of relying on dense button mappings or predefined gestures, the framework maps natural VR-controller motions in real time to vehicle pose and arm joint commands. The UAVs respond smoothly to translational and rotational inputs, while the manipulators accurately replicate dexterous hand motions for precise grasping. Beyond single-platform operation, we extend the framework to cooperative dual-UAV manipulation, leveraging two-hand poses captured via VR controllers to coordinate two UAV-arm systems for payload transportation and obstacle traversal. Simulation experiments demonstrate accurate trajectory tracking and the potential for successful cooperative transport in cluttered environments, indicating the framework’s suitability for telemanipulation, search-and-rescue, and industrial tasks.

1. Introduction

Unmanned aerial vehicles (UAVs) have been widely applied in areas such as environmental monitoring, infrastructure inspection, and logistics delivery [1]. Their adaptability arises from diverse configurations in size, structure, and control strategies, which allow them to meet different mission requirements. However, conventional UAVs mainly serve as passive observers, limited to data collection rather than direct interaction with their surroundings. To overcome this limitation, robotic arms have been integrated into UAV platforms, giving rise to aerial manipulators [2]. These systems extend UAV functionality to include physical interactions such as grasping, tool handling, and surface inspection, enabling operations in confined or hazardous environments where human presence is impractical—for example, the maintenance of bridges, wind turbines, and power lines [3].

To enhance human–robot interaction, researchers have explored more intuitive input modalities beyond traditional remote controllers. Representative approaches include gaze-based control [4,5,6], hand gestures recognized via vision or wearable devices [7,8,9,10,11,12,13], and voice commands [14]. These modalities aim to reduce operator burden, improve adaptability, and support manipulation tasks. At the same time, surveys have comprehensively reviewed teleoperation and human–robot interaction, emphasizing their importance in extending human capabilities into remote or dangerous environments [15,16,17]. Nevertheless, conventional teleoperation often depends on joysticks, screens, and button-based interfaces, which constrain spatial perception and limit the intuitiveness of control.

Recent studies have explored diverse approaches to enhance teleoperation. For instance, vision-based techniques have been employed to control manipulators using computer vision frameworks such as OpenCV [18]. In parallel, haptic feedback has been introduced to provide operators with tactile cues, improving the perception of contact and manipulation quality [19]. More recently, immersive technologies have emerged as transformative tools for intuitive teleoperation [20,21,22,23,24,25]. By embedding the operator within a real-time 3D replica of the robot’s workspace, VR offers an embodied perspective that enhances situational awareness, motor coordination, and control fluency. Such multi-sensory feedback enables operators to perform complex manipulation tasks with higher precision and reduced cognitive load. Beyond teleoperation, VR has also been integrated with deep learning approaches, where imitation learning leverages natural VR-based demonstrations to train visuomotor policies directly from pixels [26]. Collectively, these advances highlight VR as a promising paradigm for natural, efficient, and scalable teleoperation frameworks.

Despite these advances, most VR-based teleoperation frameworks remain restricted to single platforms, where UAV flight and manipulation are often handled sequentially rather than in a fully integrated manner. Interfaces typically employ a single VR controller, thereby limiting operators to one-handed control and underutilizing bimanual capability. This restriction not only reduces operational efficiency but also prevents operators from exploiting natural two-handed coordination, which is often essential for complex manipulation tasks.

At the same time, cooperative aerial transportation using multiple UAVs has been extensively investigated, including methods for collision-free trajectory planning [27], the transportation of flexible payloads [28], and the vision-based coordination of small quadrotors [29]. These works have demonstrated effective cooperative manipulation but largely depend on pre-programmed planning, strict synchronization, or communication strategies, which limit adaptability in dynamic and unstructured environments. As a result, they primarily address coordination and stability at the control level while overlooking how immersive interfaces could enable intuitive, real-time, and bimanual human control. In parallel, recent studies on shared autonomy in teleoperation [30,31] have shown that blending human input with autonomous behaviors can effectively reduce cognitive load and enhance the robustness of remote robotic manipulation. In contrast, the proposed VR framework remains fully operator-driven, focusing on reducing cognitive effort through enhanced spatial perception rather than control delegation. This distinction positions the present work as a perception-augmented alternative to shared-autonomy strategies, emphasizing intuitive human engagement over automated control sharing.

To address these limitations, this paper proposes an immersive VR-based teleoperation framework that enables a single operator to intuitively control one or two UAV–manipulator systems using dual VR controllers.

The main contributions of this work are as follows:

1

Proposed an intuitive VR-based teleoperation framework for aerial manipulation, enabling simultaneous control of a UAV and its onboard manipulator through natural hand motions captured by VR controllers, without relying on dense button mappings or predefined gestures.

2

Demonstrated the framework in single-UAV scenarios, where the operator controlled both the UAV flight and the manipulator joints using VR controllers to approach, align with, and grasp a target object in simulation.

3

Extended the framework to dual-UAV cooperative manipulation, leveraging the operator’s two-hand poses to directly map onto two UAV–manipulator units, thereby achieving coordinated payload transportation and obstacle traversal in simulation.

4

Integrated the above components into a unified teleoperation architecture that connects human motion input, UAV dynamics, and manipulator control within the same framework. This system-level integration demonstrates how existing technologies can be cohesively combined to enable intuitive, synchronized, and scalable human–robot collaboration for aerial manipulation.

Collectively, these contributions advance UAV teleoperation from single-platform control toward scalable, immersive, and cooperative multi-UAV manipulation, providing a foundation for future applications in inspection, disaster response, and industrial operations.

2. Control Methods Design

2.1. Single-UAV and Manipulator Control

Firstly, traditional quad-copter UAVs are typically operated using a handheld radio transmitter, which adjusts motor speeds to control throttle, yaw, pitch, and roll, as illustrated in Figure 1. The onboard flight controller interprets these commands to maintain flight stability and execute precise maneuvers.

Moreover, conventional setups limit one operator to control only a single UAV, making coordinated multi-UAV operations difficult and even more demanding when robotic arms are attached. To address these issues and offer more intuitive user experience, we propose a UAV control system based on a VR controller. As shown in Figure 2, the controller features a circular touchpad capable of detecting finger position. And the Mode button, Trigger button, and Gripper button each correspond to different functions. The user can simply slide their finger to move the UAV in any horizontal direction—forward, backward, left, or right within the global coordinate frame. The touch location also determines the speed of movement, allowing the operator to modulate speed naturally. The system continuously monitors the touch coordinates and, through ROS integration, translates them into real-time velocity commands. This setup supports fluid, responsive control and minimizes cognitive demand. By enabling more instinctive and proportional speed adjustments, it improves both maneuverability and user engagement, simplifying remote UAV operation overall.

The UAV’s movement involves not only horizontal translation but also adjustments in altitude and rotation around the yaw axis. As illustrated in Figure 3, altitude control is governed by the pitch angle of the VR controller. The controller’s orientation is continuously tracked in real time by a pair of positioning base stations. The resulting orientation data are transmitted from Unity (a cross-platform game engine widely used for simulation and visualization) to the main control computer via rosbridge. By interpreting the quaternion data representing the controller’s attitude, the system calculates the real-time pitch angle. When the controller is held level, no change in altitude occurs. In this setup, a tilt threshold of 60 degrees is empirically defined: pitching the controller upward increases altitude, while pitching it downward causes descent. The pitch angle of the VR controller is continuously tracked, and the corresponding upward or downward velocity commands are transmitted via ROS for the UAV’s flight controller. The corresponding control logic for altitude adjustment is outlined in Algorithm 1.

Algorithm 1 Tilt-Triggered UAV Altitude Control (Ascent/Descent)

$\mathbf{Input:}\mathrm{Controller}\mathrm{quaternion}q=\left(w,\hspace{0.17em}x,\hspace{0.17em}y,\hspace{0.17em}z\right)$$,\mathrm{threshold}{\theta }_{\mathrm{t}\mathrm{h}}={60}^{\circ }$ Output: UAV motion command trigger 1: while system is running do 2:        Read controller orientation via ROS topic $3:\mathrm{ Compute}\mathrm{pitch}\mathrm{angle} {\theta }_{\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{c}\mathrm{h}}=\arcsin \left(2\left(wy-zx\right)\right)$ $4:\mathbf{ if} {\theta }_{\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{c}\mathrm{h}}>{\theta }_{\mathrm{t}\mathrm{h}}$ then 5:            Send ascending command to UAV $6:\mathbf{ else}\mathbf{if} {\theta }_{\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{c}\mathrm{h}}<-{\theta }_{\mathrm{t}\mathrm{h}}$ then 7:               Send descending command to UAV 8:        else 9:             Maintain UAVs in hover state 10: end while

Yaw control of the UAV follows a similar principle to altitude control. As shown in Figure 4, changes in yaw are governed by the roll angle of the VR controller. When the controller is held level along the roll axis, the UAV maintains its current yaw orientation. A tilt threshold of 60 degrees is defined for this axis as well: tilting the controller to the left results in a counterclockwise yaw rotation of the UAV, whereas tilting it to the right produces a clockwise rotation. The controller’s roll angle is continuously monitored and extracted from its quaternion-based attitude data. These real-time roll values are transmitted via ROS and converted into appropriate yaw velocity commands for the UAV’s flight controller.

The use of a VR controller for manipulator control represents a significant advancement in robotic teleoperation, providing an intuitive interface that directly maps the operator’s intended motions to the system. This section describes the mechanisms through which the VR controller regulates the spatial position and orientation of the manipulator’s end-effector. As illustrated in Figure 5, the manipulator mounted on the UAV platform is equipped with five degrees of freedom (DoF), denoted as joints J1 through J5. While not every joint is directly and independently controlled by the VR device, the control scheme emphasizes intuitive mapping of the controller’s motion to the task space, with certain joints coordinated automatically to realize the commanded end-effector pose. Specifically, J1–J3 correspond to the main arm joints that primarily determine the gross positioning of the manipulator, J4 and J5 function as wrist joints enabling fine orientation adjustments, and the final gripper joint provides grasping capability for object interaction. The following subsections explain the functional roles of J1–J5 and how the VR controller inputs are assigned to these DoFs.

The first three degrees of freedom of the robotic manipulator—J1 through J3—are responsible for determining the approximate spatial position of the end-effector. Utilizing the spatial tracking capabilities of the VR controller, the system captures the operator’s hand position in three-dimensional space and transmits this data to the MoveIt motion planning framework on the master computer via rosbridge. MoveIt is a widely adopted motion planning tool, particularly effective for managing complex motion sequences and path planning in robotic systems. Based on the received spatial coordinates, MoveIt performs inverse kinematics (IK) computations to generate the corresponding joint commands, which drive the actuators associated with J1, J2, and J3.

By holding the Mode button and simply moving the VR controller in free space, the operator can intuitively control the position of the manipulator’s end-effector. This direct mapping between human movement and robot response enhances positioning accuracy, reduces the operator’s cognitive workload, and facilitates the execution of sophisticated manipulation tasks.

In addition to position control, the system also supports rotational manipulation of the end-effector, specifically the wrist pitch (J4) and wrist roll (J5). These rotational degrees of freedom are adjusted by modifying the orientation of the VR controller, namely its pitch and roll angles. To distinguish manipulator control from UAV control, a mode-switching mechanism is implemented using a button on the VR controller. When the button is held, the system enters manipulator control mode, allowing changes in orientation to affect only the manipulator. Meanwhile, by holding the Trigger button, the user can control the two wrist joints (J4, J5) simultaneously. Once the Mode button is released, the system returns to UAV control mode. This switching mechanism effectively prevents control conflicts and enhances the overall operability of the system.

As illustrated in Figure 3, the pitch axis of the VR controller is mapped to joint J4, which governs the pitch of the manipulator’s wrist. Similarly to how altitude is controlled in the UAV system, the operator can intuitively increase or decrease the wrist pitch angle by tilting the VR controller upward or downward. Figure 4 shows the control of J5, corresponding to the roll axis of the manipulator’s wrist. In the same manner as yaw control for the UAV, the operator can rotate the VR controller along its roll axis to adjust the wrist’s roll angle in either direction. This direct mapping between the controller’s orientation and the manipulator’s rotational movement facilitates precise and intuitive wrist control during task execution.

In summary, the proposed control method leverages natural human motion, captured through the VR controller, to provide an intuitive and highly responsive interface for operating the robotic manipulator. This approach not only simplifies control in both industrial and research contexts but also broadens the scope for remote and precise manipulation in a range of application domains. The seamless integration of spatial tracking with real-time motion mapping enhances operability, making complex tasks more accessible.

2.2. Dual-UAV Cooperative Control

In cooperative aerial manipulation, two UAVs are often required to transport elongated payloads through constrained environments while maintaining stability and controllability. This introduces challenges that are not encountered in single-UAV tasks, such as coordinating relative motion, ensuring continuous visual feedback, and adapting to obstacles or gaps of varying geometry. To address these requirements, we designed a set of fundamental operation modes that can be intuitively commanded through VR-based hand gestures. All gestures, even in the case of dual-UAV cooperative control, do not directly correspond to the real-time motion of the drones or manipulators, but rather serve as triggers to initiate or terminate each operation.

Figure 6 illustrates these operation modes. In (a)-(1), translational gestures are mapped to UAV movements in six directions within the global coordinate frame, enabling the operator to reposition the vehicles in three-dimensional space. This capability is crucial for navigation in cluttered environments, where frequent and precise positional adjustments are required. In (a)-(2) and (3), the yaw motion of the UAV is coupled with the first manipulator joint (J1). Since the onboard camera is mounted at the front of the UAV, yawing alone would cause the camera to turn away from the direction of travel during lateral motion. By coupling yaw with the first joint, the manipulator automatically rotates in the opposite direction, keeping the payload aligned with the moving direction and maintaining continuous visual feedback. Subfigures (a)-(4) and (5) show circular gestures, where the two hands are positioned sequentially with a phase offset, resulting in coordinated circular trajectories. This design supports cooperative reorientation and encirclement of elongated payloads, as circular motion provides a predictable and stable relative geometry between the UAVs. Finally, (a)-(6) and (7) demonstrate inclined-surface traversal, in which one UAV maintains a higher altitude and the other a lower altitude, causing the payload to tilt. This mode was introduced to allow passage through narrow or slanted gaps, which are representative of realistic operational constraints.

The symmetric circular motion of the dual-UAV system is modeled with respect to a common center $\mathit{c}=\left(x_{\mathrm{c}},y_{\mathrm{c}}\right)$. The leader UAV1 follows a circular trajectory parameterized by its phase angle $\varphi \left(t\right)$, while the follower UAV 2 remains at the diametrically opposite position with respect to the center. This configuration keeps the UAVs evenly distributed on the circle and ensures a constant inter-UAV distance, thereby guaranteeing non-intersecting trajectories during cooperative maneuvers.

The phase angle $\varphi \left(t\right)$ evolves as:

$$\varphi \left(t\right)={\varphi }_0+\omega t.$$

(1)

The positions of the UAVs are expressed as:

$${\mathit{p}}_1\left(t\right)=\left[\begin{array}{c}{x}_{\mathrm{c}}+r\mathrm{c}\mathrm{o}\mathrm{s}\left(\varphi \left(t\right)\right)\\ y_{\mathrm{c}}+r\mathrm{s}\mathrm{i}\mathrm{n}\left(\varphi \left(t\right)\right)\\ h_1\end{array}\right],\hspace{1em}{\mathit{p}}_2\left(t\right)=\left[\begin{array}{c}{x}_{\mathrm{c}}-r\mathrm{c}\mathrm{o}\mathrm{s}\left(\varphi \left(t\right)\right)\\ y_{\mathrm{c}}-r\mathrm{s}\mathrm{i}\mathrm{n}\left(\varphi \left(t\right)\right)\\ h_2\end{array}\right].$$

(2)

Yaw orientations are calculated as:

$${\psi }_1\left(t\right)=\mathrm{a}\mathrm{t}\mathrm{a}\mathrm{n}2\left(\dot{y_1},\dot{x_1}\right)=\varphi \left(t\right)+{\displaystyle \frac{\mathsf{\pi }}{2}},{\psi }_2\left(t\right)={\psi }_1\left(t\right)+\mathsf{\pi }.$$

(3)

To translate the VR controller input into this circular motion behavior, the real-time positions of the two controllers are interpreted to determine the rotational direction and trigger corresponding UAV maneuvers. The detailed procedure of this dual-UAV rotation control process is summarized in Algorithm 2 (Dual-UAV Rotation Control via VR Input).

Algorithm 2 Dual-UAV Rotation Control via VR Input

Input: Positions of left and right VR controllers $\mathrm{ }{p}_L=\left(x_L,y_L,z_L\right),\hspace{1em}{p}_R=\left(x_R,y_R,z_R\right)$ Output: Direction of circular maneuver (CW/CCW) 1: while system is running do 2:        Read controller positions via ROS topic $3:\mathrm{ Compute}\mathrm{position}\mathrm{difference} \Delta y=y_R-y_L$ $4:\mathbf{ if} \Delta y$ > 0.30 m then 5:            Trigger counter-clockwise (CCW), set ω > 0 $6:\mathbf{ else}\mathbf{if} \Delta y$ < −0.30 m then 7:             Trigger clockwise (CW), set ω < 0 8:        else 9:             Maintain UAVs in hover state 10: end while

As illustrated in Figure 7, this formulation ensures that the UAVs remain symmetrically opposed on the circle with consistent tangential orientation, enabling cooperative encirclement maneuvers.

In the yaw compensation of UAV–manipulator configuration, when a UAV carries a robotic manipulator, the manipulator base is mechanically fixed to the UAV body frame $\left\{\mathrm{B}\right\}$, while the world reference frame is denoted as $\left\{\mathrm{W}\right\}$. The yaw angle of the UAV with respect to the world frame is expressed as ${\psi }_d\left(t\right)$. The first revolute joint of the manipulator, denoted as J1, rotates around the vertical axis with angle ${\theta }_1\left(t\right)$. Although the manipulator is structurally fixed to the UAV, its effective yaw orientation in the world frame can be decoupled by appropriately controlling J1.

Without such compensation, any change in the UAV yaw angle directly alters the orientation of the manipulator in $\left\{\mathrm{W}\right\}$, leading to undesired motion of the end-effector. To address this issue, we introduce a yaw compensation mechanism at joint J1.

The UAV yaw rotation can be described by the planar rotation matrix:

$${\mathrm{R}}_z\left({\psi }_d\right)=\left[\begin{array}{cc}\cos {\psi }_d& -\sin {\psi }_d\\ \sin {\psi }_d& \cos {\psi }_d\end{array}\right],$$

(4)

where ${\psi }_d\left(t\right)$ is the UAV yaw angle in the world frame.

The absolute yaw orientation of the manipulator base in the world frame, denoted as ${\psi }_m\left(t\right)$ is given by:

$${\psi }_m\left(t\right)={\psi }_d\left(t\right)+{\theta }_1\left(t\right),$$

(5)

where ${\theta }_1\left(t\right)$ is the rotation of the first manipulator joint about the vertical (yaw) axis.

To ensure that the manipulator maintains constant yaw in the world frame, i.e.,

The corresponding compensation law is derived as:

$${\theta }_1\left(t\right)={\psi }_m\left(0\right)-{\psi }_d\left(t\right).$$

(6)

Accordingly, the detailed control logic of the VR-Triggered Yaw Compensation for UAV–Manipulator System is summarized in Algorithm 3.

Algorithm 3 Yaw Compensation for UAV–Manipulator System via VR Input

Input: Rotations of left and right VR controllers $\mathrm{ }{q}_R=\left(w_R,x_R,y_R,z_R\right)$ Output: Yaw rotation of the UAV and inverse compensation of joint J1 1: while system is running do 2:        Read controller rotations via ROS topic $3:\mathbf{}\mathrm{ Extract}\mathrm{roll}\mathrm{angle}\mathrm{of}\mathrm{the}\mathrm{right}\mathrm{controller} {\varphi }_R$$\mathrm{from}\mathrm{quaternion} q_R$ $4:\mathbf{ if} {\varphi }_R\ge {90}^{\circ }$ and left controller remains level then $5:\mathrm{ Set}\mathrm{UAV}\mathrm{yaw} {\psi }_d={\psi }_c+{\mathsf{\Delta }}_{\mathrm{offset}}$ $6:\mathrm{ Set}\mathrm{arm}\mathrm{joint}\mathrm{J}1\mathrm{angle} {\theta }_1=-\hspace{0.17em}{\psi }_c$ 7:        else 8:              Maintain UAVs in hover state 9: end while

This compensation mechanism allows the UAV to freely perform yaw maneuvers for navigation or perception purposes, while the manipulator maintains a fixed orientation in the global frame, as illustrated in Figure 8. Such a principle is essential for tasks requiring stationary alignment or precise environmental interaction, such as grasping, docking, or tool operation.

During cooperative manipulation, the two UAVs may intentionally adopt different altitudes to traverse inclined planes or perform asymmetric maneuvers. When the altitude difference between the UAVs is $\Delta h$ and their horizontal separation is $d$, the manipulated rod will form an inclination angle $\theta$ with respect to the horizontal plane, as illustrated in Figure 9. Since each gripper is rigidly attached to the rod, misalignment between the rod and the end-effector orientation would introduce undesirable torques and mechanical stress. To mitigate this, one degree of freedom of the manipulator wrist (the roll joint) is dedicated to compensating for the rod inclination. Accordingly, the control logic of the tilt-triggered motion and wrist compensation is described in Algorithm 4.

Algorithm 4 Cooperative UAV Motion and Wrist Compensation via VR Input

Input: Positions of left and right VR controllers $\mathrm{ }{p}_L=\left(x_L,y_L,z_L\right),\hspace{1em}{p}_R=\left(x_R,y_R,z_R\right)$ Output: Vertical motion of UAVs and J5 compensation angles 1: while system is running do 2:        Read controller positions via ROS topic $3:\mathbf{}\mathrm{ Compute}\mathrm{position}\mathrm{difference} \Delta z=z_L-z_R$ $4:\mathbf{ if} \mid \mathsf{\Delta }z\mid >0.20\hspace{0.17em}\mathrm{m}$then $5:\mathbf{ if} \mathsf{\Delta }z>0$ then 6:                  Left UAV ascends; Right UAV descends. 7:             else 8:                  Right UAV ascends; Left UAV descends. 9:             Compute inclination angle $10:\mathrm{ }\theta =\arctan \left({\displaystyle \frac{\Delta h}{d}}\right)$ 11:           Set commanded wrist roll angles: $12:\mathrm{ }{q}_{\left\{\mathrm{r}\mathrm{o}\mathrm{l}\mathrm{l},\mathrm{c}\mathrm{m}\mathrm{d}\right\},1}=q_{\left\{\mathrm{r}\mathrm{o}\mathrm{l}\mathrm{l},\mathrm{c}\mathrm{m}\mathrm{d}\right\},2}=q_{\left\{\mathrm{r}\mathrm{o}\mathrm{l}\mathrm{l},\mathrm{r}\mathrm{e}\mathrm{f}\right\}}+\theta$ 13:        else 14:           Maintain UAVs and manipulators in hover state. 15: end while

This compensation law ensures that the gripper orientation remains aligned with the rod axis, regardless of UAV altitude differences. Consequently, the UAVs can stably manipulate elongated objects or traverse inclined planes without inducing excessive mechanical load. Moreover, this strategy effectively decouples manipulator alignment from UAV flight control, improving stability and robustness of cooperative aerial manipulation.

In summary, the proposed Dual-UAV Cooperative Control establishes a unified scheme that enables two UAV-arm systems to function as a coordinated unit. By integrating VR-based teleoperation with cooperative control strategies, the framework achieves synchronized grasping, maintains consistent inter-vehicle geometry, and demonstrates stable cooperative transport in simulation. These results provide a methodological basis for extending aerial manipulation from single-UAV control toward dual-UAV cooperation, thereby preparing the ground for the experimental validation and broader multi-robot applications discussed in the following sections.

3. Experiments Using VR System

3.1. Experimental Setup

The control system (Figure 10) consists of a VR interface on a Windows PC and a simulation environment with ROS on an Ubuntu PC. The VR interface includes an HMD and controllers, while Unity and rosbridge handle communication with the simulation. The environment contains multiple UAVs equipped with manipulators and cameras for aerial grasping tasks. The software integrates ROS, Gazebo, Unity, and rosbridge. ROS manages communication, sensor fusion, and motion planning; Gazebo provides physics-based simulation; Unity offers real-time visualization; and rosbridge enables data exchange. A dual-PC configuration is employed, with ROS, Gazebo, and rosbridge running on Linux and Unity on Windows, to reduce latency by separating VR rendering from physics simulation and ensuring synchronized teleoperation. To clarify the simulation settings, the flight characteristics of the quadcopters used in the experiments are listed in

Table 1. UAV Parameters.

Mass (kg)	4.0
Moment of Inertia (kg·m2)	diag (0.072, 0.135, 0.153)
Wheelbase (m)	1.3
Thrust Coefficient (N/rpm2)	3.0 × 10−7
Moment Coefficient (Nm/rpm2)	4.0 × 10−8

.

As illustrated in Figure 11, the VR kit (HTC Corporation, Taoyuan City, Taiwan) includes an HMD, controllers, and positioning base stations. By tracking the controllers with two base stations, the system captures spatial pose and maps hand motions to UAV flight and manipulator commands, enabling immersive and natural interaction with the simulated environment.

3.2. Experimental Tasks

To evaluate the effectiveness of the proposed VR-based control framework, a series of experimental tasks were designed in both single-UAV and dual-UAV scenarios. The tasks were selected to reflect representative challenges in aerial manipulation, including precise positioning, cooperative control, and operation in constrained environments.

In the task shown in Figure 12, a single UAV equipped with a manipulator was required to approach a designated target object and perform a grasping action. This task validates the feasibility of using the VR interface for integrated control of both UAV flight and manipulator operation, while ensuring accurate positioning of the end-effector in cluttered environments.

In the next experiment, a cooperative transportation task was simulated, in which two UAVs carried an elongated object together. The operation began with the UAVs tilting to traverse an inclined surface while carrying the object. They then executed a coordinated circular maneuver to demonstrate rotational motion. Next, the leader UAV adjusted its heading to release the object at the designated position, and finally both UAVs exited the environment through a narrow opening. Due to the inherent physical properties of the simulator, the grasped object tended to gradually slip after a certain period of time. Therefore, at the current stage, each phase was conducted and validated separately and subsequently linked to emulate the achievable behavior in real-world conditions, as shown in Figure 13. In addition, the operator performed all actions under both first-person and third-person perspectives to ensure accurate situational awareness and control during the experiment. This experiment demonstrates the capability of the VR-based interface to support complex cooperative aerial transport and highlights the challenges of maintaining coordination, stability, and precise control in constrained environments.

3.3. Experimental Results

The proposed framework was first validated in a single-UAV scenario to examine its capability for precise teleoperation. In this setting, the UAV was required to approach a target object, maintain a stable hovering position, and operate its onboard manipulator to accomplish grasping tasks. To preliminarily test the accuracy of the end-effector control, the manipulator was commanded to trace predefined geometric shapes, including a square and a triangle. The resulting trajectories (Figure 14) clearly demonstrate smooth and repeatable paths, indicating that the VR controller provided sufficient precision for guiding the end-effector along complex spatial patterns.

In addition to manipulator control, the UAV’s overall flight stability was also examined, Figure 15 reconstructs the spatial trajectory. Complementary snapshots (Figure 16, arranged from top to bottom and left to right) illustrate the sequence of the flight and grasping process: the UAV first approaches the target, then switches to the grasping mode to secure the object, and finally increases altitude to depart from the grasping area. These results demonstrate that the proposed framework enables intuitive and coordinated control of UAV flight and manipulator motion.

The proposed VR-based teleoperation framework was further validated in dual-UAV cooperative experiments involving obstacle traversal. In this task, the UAVs carried an elongated object through narrow passages, placed it at a designated location, and then departed while maintaining formation flight. This sequence demonstrates the ability of the framework to coordinate transport, placement, and withdrawal under constrained conditions. Figure 17 (2D top view) and Figure 18 (3D perspective) illustrate the reconstructed path that could be ideally achieved under optimal conditions. While the sequential snapshots in Figure 19 (arranged from top to bottom and left to right) illustrate the detailed motion process under VR teleoperation.

Quantitative analyses further confirm the effectiveness of the proposed framework for coordinating UAV–manipulator systems. Figure 20 compares the UAV inclination angle with the wrist roll joint, showing that the compensation strategy aligned the manipulator with the rod during inclined transport and reduced unwanted torsional effects. In Figure 21, the yaw trajectory is plotted against the commanded reference, indicating accurate heading adjustment throughout the maneuver. The results on Figure 22 demonstrate that both UAVs exhibited synchronized yaw changes (a) while keeping a nearly constant inter-UAV distance of about 4 m (b). This stable relative positioning suggests that the two platforms remained coordinated without noticeable drift. Overall, the findings highlight the feasibility of the compensation approach and its robustness in maintaining geometric constraints, enabling reliable dual-UAV cooperation even under inclination and yaw disturbances.

4. Discussion

The experimental results confirmed that the proposed method works for both single- and dual-UAV manipulation tasks. However, several limitations were observed. First, the current study was conducted in a simulation environment without external disturbances such as wind or actuator faults; hence, explicit compensation or fault-tolerant mechanisms were not implemented. To improve stability in practical scenarios, several countermeasures can be considered. These include incorporating disturbance observers or adaptive control strategies to mitigate the effects of crosswind and adding a passive vibration absorption mechanism to the manipulator to reduce small oscillations transmitted from the UAV body. Furthermore, basic fault detection and recovery strategies could be introduced to protect the system in case of rotor malfunction or unexpected disturbances.

In addition, we acknowledge that certain failure cases may occur under conditions of poor tracking or high visual clutter, which could lead to delayed or inaccurate perception of spatial references. To address this, future work will explore robust sensor fusion and adaptive visual filtering techniques to maintain stable perception in such visually challenging environments.

Finally, the present work focuses on a dual-UAV–manipulator configuration based on rotorcraft platforms; the proposed framework is inherently scalable to multi-UAV cooperative scenarios for handling heavier payloads. The VR-based teleoperation interface and symmetric control mapping are platform-agnostic, emphasizing intuitive human–robot interaction rather than vehicle-specific dynamics. Consequently, parts of the proposed method can also be adapted to fixed-wing UAVs by modifying the motion mapping layer to account for flight constraints, while the operator interface can remain unchanged. Moreover, the same modular structure allows extension to heterogeneous aerial or ground robots through unified communication layers. Future work will explore these generalizations to larger robot teams and dynamic environments, aiming to enhance robustness, coordination, and shared autonomy.

5. Conclusions

In this paper, we proposed a novel control scheme for single and dual manipulator UAV operation by a single operator through a VR-based system. The proposed framework integrates UAV flight and manipulator control into a unified interface, allowing the operator to directly perform complex spatial tasks using natural hand motions. To verify the feasibility of this approach, we conducted simulation experiments in which the operator, equipped VR controllers, successfully manipulated one or two UAVs to perform tasks. The results demonstrated that coordinated control of UAV flight and manipulator motion can be achieved in real time, and that a single operator can intuitively manage dual-UAV cooperative manipulation. These findings confirm the effectiveness of the VR-based control system in enabling seamless interaction between the operator and UAV–manipulator platforms and highlight its potential for tasks requiring dexterous and cooperative aerial manipulation.

In future work, we will further improve our system to experimentally validate a coherent transportation task in a first-person teleoperation setting and extend it from simulation to real-world experiments using UAV platforms equipped with manipulators. Moreover, a comparative evaluation with conventional control approaches will be carried out to quantitatively assess task completion time, success rate, and overall system performance. In addition, an ergonomic evaluation will be performed to analyze the operator’s comfort, workload, and motion efficiency. This evaluation will include both subjective measures, such as perceived fatigue and usability questionnaires, and objective indicators, such as motion trajectories and task execution smoothness, to comprehensively assess human–system interaction during extended teleoperation. Furthermore, the collected results will help identify design improvements in the teleoperation interface and control mapping, contributing to a more intuitive and fatigue-resistant user experience. Finally, applying the system to practical scenarios—such as cooperative inspection, aerial assembly, or disaster response—will be an important step toward demonstrating its usefulness outside the laboratory.