Hovering of Bi-Directional Motor Driven Flapping Wing Micro Aerial Vehicle Based on Deep Reinforcement Learning

Abstract

Inspired by hummingbirds and certain insects, flapping wing micro aerial vehicles (FWMAVs) exhibit potential energy efficiency and maneuverability advantages. Among them, the bi-directional motor-driven tailless FWMAV with simple structure prevails in research, but it requires active pose control for hovering. In this paper, we employ deep reinforcement learning to train a low-level hovering strategy that directly maps the drone’s state to motor voltage outputs. To our knowledge, other FWMAVs in both reality and simulations still rely on classical proportional-derivative controllers for pose control. Our learning-based approach enhances strategy robustness through domain randomization, eliminating the need for manually fine-tuning gain parameters. The effectiveness of the strategy is validated in a high-fidelity simulation environment, showing that for an FWMAV with a wingspan of approximately 200 mm, the center of mass is maintained within a 20 mm radius during hovering. Furthermore, the strategy is utilized to demonstrate point-to-point flight, trajectory tracking, and controlled flight of multiple drones.

1. Introduction

Bees and certain other insects are capable of hovering, while hummingbirds are the only bird species capable of such sustained aerial behavior [1]. Mimicing the flight of these biological flyers, flapping wing micro aerial vehicles (FWMAVs) is an attractive solution for drones at the scale of centimeters or decimeters. At this scale, fixed wings struggle to maintain high speeds for sufficient lift, and rotor-based forward flight suffers from inadequate energy efficiency [2]. In contrast, flapping wing designs hold promise for maneuverability within confined spaces and greater energy efficiency [3,4]. However, as of now, due to manufacturing flaws, insufficient motor power density, limited control methods, and other reasons [5,6], FWMAVs still struggle to emulate their counterparts in nature.

In the past few decades, several research teams have achieved liftoff and to some extent controlled flight of FWMAVs. Typically, researchers utilize crank-rocker mechanisms to convert the unidirectional rotation output of motors into flapping motion of wings, as demonstrated in projects such as Delfly and KUBeetle [7,8,9]. One limitation of this type of aircraft is its inability to alter the flapping amplitude of wings, only being able to adjust the flapping frequency, necessitating additional servo mechanisms to generate control torque. Other researchers have employed high-frequency input power modulation, driving wing flapping directly through piezoelectric or bidirectional motor rotation, as seen in projects like Robobee and Purdue hummingbird robot [10,11,12]. This approach allows for easy manipulation of wing kinematics, controlling the left and right flapping angles of wings independently, thus generating control torque through instantaneous wing kinematics modulation [13], without the need for additional mechanisms for control assistance.

The flight control of FWMAVs presents significant challenges, including complex system dynamics, severe underactuation, and limited control authority, among others [2,14]. Some researchers address these issues using classical feedback controllers. Karásek et al. employ a biologically inspired Proportional-Derivative (PD) controller for pitch and roll control of the Delfly Nimble, and a proportional and feedforward controller for yaw control [7]. Phan et al. use a PD controller for feedback control of the KUBeetle’s angular velocity, with three servos used to minimize the error between the desired and measured bodyrates [15]. These fixed-gain feedback architecture heavily relies on empirical engineering practices, involving trial and error for gain parameter tuning in both simulation and real-world scenarios [2,15]. As for FWMAVs without crank-rocker, Chirarattananon et al. propose a linear controller by sequentially adjusting parameters to accomplish increasingly complex control tasks [16]. Zhang et al. design a nonlinear geometric controller that theoretically ensures it is almost globally exponentially attractive and achieves attitude stabilization control during the liftoff process [14].

Researchers also attempt flight control of FWMAVs through learning-based methods. Chirarattananon et al. propose an iterative learning control algorithm, building upon the Robobee platform, to achieve imitation of insect-like landing on vertical walls with the aid of magnetic adhesion [17]. Fei et al. employ a hybrid strategy, combining adaptive robust control with the Deep Deterministic Policy Gradient (DDPG) algorithm, to enable FWMAVs to execute aggressive maneuvers like 180-degree yaw turns and 360-degree body rolls [18,19]. Nozawa et al. utilize the Deep Q-Network algorithm to control the pitch angle of a FWMAV [20], yet this control is discretized, capable only of maintaining the pitch angle within a certain range. In fact, the aforementioned studies only use learning-based methods to perform some specific tasks, while relying on classical feedback controllers to achieve pose control tasks such as hovering.

The control issues of FWMAVs may benefit from insights gained through training quadcopter flight using learning-based method. While many studies focus on high-level decision-making [21,22], particularly trajectory planning, there are still some works utilizing learning-based methods to perform low-level control. These works directly output motor voltage commands for pose control, thereby eliminating the need for an additional classical lower-level pose controller. Koch et al. propose a quadrotor low-level controller based on reinforcement learning which can track the set values of angular velocity [23]. Molchanov et al. train a stable low-level quadrotor controller that can drive the Z-axis of the drone’s body coordinates vertically upward and minimize position and velocity errors from randomly sampled initial conditions within a certain range in a short time [24].

In this paper, we choose a hummingbird-like bi-directional motor-driven tailless FMWAV as the subject due to its simplicity, compactness, and potential maneuverability. The goal is to develop a hover flight strategy capable of consistently maintaining the center of mass of the FWMAV’s torso within a 50 mm diameter space during flight. Considering the wingspan of this kind of FWMAV is about 200 mm, this positional error is acceptable. The experiments are performed in a high-fidelity simulation environment, which has complete system dynamics, including motor-driven wing dynamics and flapping flight aerodynamics. We use reinforcement learning to train the hover strategy. The training is conducted in 20 parallel environments, employing the widely-used Proximal Policy Optimization (PPO) algorithm to ensure stability in policy updates. Reasonable state and action spaces are selected, with particular emphasis placed on utilizing a sliding window of historical states and actions to expand the state space, addressing asynchronous control issues [25]. Boundary conditions and reward functions are designed through understanding of the problem and iterative refinement. The final strategy is obtained through two training sessions, each consisting of 50 million steps, with different boundary conditions set for each training session. During the first training session, with larger boundary values, the FWMAV primarily learns to maintain attitude stability. In the second training session, boundary values are reduced to enhance the precision of the position control.

Eventually, the trained policy can enable the FWMAV to hover at any set point in space and additionally perform vertical, forward, and lateral flights, enabling point-to-point flight from the initial position to distant target points. This policy effectively serves as a low-level pose controller, while avoiding manual tuning of complex gain parameters. To our knowledge, other FWMAVs in reality or simulation still rely on pose controllers similar to traditional cascaded PD. As a demonstration of the policy’s versatility, trajectory tracking along a circular path and the control of multiple drones are shown within the simulation.

The structure of this paper is as follows: Section 2 elaborates on the modeling of FWMAVs and briefly introduces reinforcement learning method. Section 3 provides a detailed description of the experimental intricacies of reinforcement learning training, alongside the measurement of forces on the FWMAV in simulation. Section 4 presents a analysis of training results, supplemented with demonstrations of several applications. Finally, Section 5 concludes and points out future research directions.

2. Models and Methods

In this chapter, the fundamental model of our FWMAV is established, encompassing the mechanical structure, wing actuation model, and aerodynamics. Subsequently, we analyze the generation of control forces and torques, then outline the control challenges of this type of drone. Finally, the reinforcement learning algorithm used for continuous action control in this paper is briefly introduced.

2.1. Drone Model and Flapping Mechanisms

The bi-directional motor driven tailless FWMAV prototype implemented in our lab is shown in Figure 1a. Corresponding SolidWorks model is shown in Figure 1b, where a hollow carbon fiber frame is employed as the torso and two rigid thin sheets as the wings. Note that the torso is vertical here, an orientation resembling that of a hummingbird during flight, where it typically maintains a slight angle with the vertical direction, as opposed to large-scale flapping-wing aerial vehicles or fixed-wing aerial vehicles.

Two brushless direct current motors are employed to drive the wings separately, enabling a symmetric flapping motion between the upstroke and downstroke phases through gearsets and torsion springs. In addition, the wings undergo passive rotation about the rotation axis due to forced vibration. Elastic components can function as energy storage elements to preserve the kinetic energy, and a properly configured flapping frequency can induce mechanical resonance, leading to a significant enhancement in the energy efficiency of the entire system [12,26]. In an ideal situation, the kinetic and potential energies of the mechanical components within the system can be conserved, with the majority of the motors’ work dedicating to overcoming system damping forces and aerodynamic drag on the wings, without being utilized to counteract the inertia force resulting from the high-frequency reciprocating motion of the wings [27].

To reduce the complexity in simulation, the Unified Robot Description Format(URDF) model is simplified into five components: torso, two rods, and two wings, as shown in Figure 1c. The mass and rotational inertia are systematically identified in reality and then injected into the urdf file. The body coordinate system OXYZ is fixed on the torso. The $X^{\prime }{Y}^{\prime }{Z}^{\prime }$ coordinate system is at the right wing root. The $Y^{\prime }$ axis points radially outward, and the $Z^{\prime }$ axis points opposite to the chordwise direction of the wing.

The wing can rotate around the rotation axis, with the rotation angle denoted as ${\psi }_w$. The T-shaped rod, swinging together with the wing up and down around the stroke axis, is referred to as the stroke motion, where the stroke angle is denoted as ${\varphi }_w$. The wing actuation model can be expressed as follows [27]:

$$K_{u}u=J_s{\ddot{\varphi }}_w+B_{s1}{\dot{\varphi }}_w+B_{s2}\left|{\dot{\varphi }}_w\right|{\dot{\varphi }}_w+K_s{\varphi }_w$$

(1)

where u signifies the input voltage, $K_u$ is the lumped input gain, $J_s$ denotes the total moment of inertia, $B_{s1}$ and $B_{s2}$ stand for the combined linear and aerodynamic damping coefficients respectively, $K_s$ represents the torsional spring coefficient.

Calculating aerodynamic forces for FWAMVs typically assumes that these forces are computed under quasi-steady aerodynamic conditions [28]. Transient aerodynamic phenomena such as delayed stall, rotational circulation, and wake capture [29] in low Reynolds number flight scenarios are indirectly expressed through the aerodynamic coefficient $C_F\left(\alpha \right)$, where $\alpha$ denotes the angle of attack of the wing.

To ensure clarity in the following aerodynamic discussions, we adopt Ellington’s proposed insect wing parameters [30] to model the wing, as illustrated in Figure 2. In this model, r is the radial distance, $c\left(r\right)$ denotes the chord length, R is the wing span, and S stands for the wing area. The mean chord length is calculated as $\overline{c}=S/R$. Physical quantities with a hat ^ are dimensionless, such as ${\widehat{r}}_k$ represents the dimensionless radial distance of the k-th order central moment: ${\widehat{r}}_k={\left({\int }_0^1{\widehat{r}}^k\widehat{c}\left(\widehat{r}\right)d\widehat{r}\right)}^{1/k}$, where $\widehat{r}=r/R$.

The aerodynamic forces are calculated using Blade Element Theory (BET). Considering a small chordwise strip of the wing located at a distance r from the wing root with an area $\mathrm{d}S=c\left(r\right)dr$, the relative airflow velocity is $V={\dot{\varphi }}_{w}r$. For hovering or low-speed flight, the velocity of the torso is relatively small and can be neglected. Dynamic air pressure is $q=(1/2){\rho }_{\mathrm{air}}{V}^2$. From the general form of aerodynamic force $F=C_F\left(\alpha \right)qS$, the differential form and integral form with dimensionless parameters can be expressed as follows:

$$\left\{\begin{array}{c}\begin{array}{cc}\hfill \mathrm{d}{F}_{\mathrm{aero}}& =C_F\left(\alpha \right)(1/2){\rho }_{\mathrm{air}}{\left({\dot{\varphi }}_{w}r\right)}^{2}c\left(r\right)\mathrm{d}r\hfill \\ \hfill F_{\mathrm{aero}}& ={\displaystyle \frac{1}{2}}{\rho }_{\mathrm{air}}{\dot{\varphi }}_w^2{C}_F\left(\alpha \right)\overline{c}{R}^3{\widehat{r}}_2^2\hfill \end{array}\hfill \end{array}\right.$$

(2)

In the simulation, only the normal force $F_N$ on the wing surface is considered because it far exceeds the shear viscous forces acting parallel to the wing [29]. We have:

$$F_N={\displaystyle \frac{1}{2}}{\rho }_{\mathrm{air}}{\dot{\varphi }}_w^2{C}_N\left(\alpha \right)\overline{c}{R}^3{\widehat{r}}_2^2$$

(3)

where the empirical formula for $C_N\left(\alpha \right)$ can be found in [28]. By applying the spatial resultant moment theorem, the point of application of force $F_N$ on the wing surface can be determined, namely $Y_{cp}^{\prime }=M_{Z^{\prime }}/F_N$ and $Z_{cp}^{\prime }=M_{Y^{\prime }}/F_N$. $M_Y^{\prime }$ and $M_Z^{\prime }$ are the moments of the aerodynamic force in the normal direction of the wing surface about the Y-axis and Z-axis, respectively. They can both be calculated using the aforementioned BET approach [28]:

$$\left\{\begin{array}{c}\begin{array}{cc}\hfill M_{Y^{\prime }}& ={\displaystyle \frac{1}{2}}{\rho }_{\mathrm{air}}{\dot{\varphi }}_w^2{C}_N\left(\alpha \right)d_{cp}{\overline{c}}^2{R}^3{\widehat{r}}_2^2\hfill \\ \hfill M_{Z^{\prime }}& ={\displaystyle \frac{1}{2}}{\rho }_{\mathrm{air}}{\dot{\varphi }}_w^2{C}_N\left(\alpha \right)\overline{c}{R}^4{\widehat{r}}_3^3\hfill \end{array}\hfill \end{array}\right.$$

(4)

where the $d_{cp}$ denotes the chordwise aerodynamic center coefficient [31].

As previously mentioned, while the motor actively drives the wing to perform the stroke motion, the wing undergoes passive rotation. This effect is described using rotational aerodynamic damping $M_{Y^{\prime },rd}$ [28]:

$$M_{Y^{\prime },rd}={\displaystyle \frac{1}{8}}{\rho }_{\mathrm{air}}{C}_{rd}{\dot{\psi }}_w\left|{\dot{\psi }}_w\right|{\overline{c}}^{-4}R{\int }_0^1\widehat{c}{\left(\widehat{r}\right)}^{4}d\widehat{r}$$

(5)

2.2. Generation of Control Force and Torque

In flight control systems, the concept of averaging theory is widely applied [10,11], allowing the approximation of time-varying systems through their averaged values, provided there is sufficient separation of time constants [32]. In FWMAVs, as long as the flapping frequency f is sufficiently high, an averaged system can be used through a cycle-averaged froce coefficient ${\overline{C}}_F$ [29]. Specifically, if the amplitude of the stroke angle is ${\mathrm{\Phi }}_m$, the average lift ${\overline{F}}_L$ of a single wing can be represented as:

$${\overline{F}}_L\left({\mathrm{\Phi }}_m\right)={\displaystyle \frac{1}{2}}{\rho }_{\mathrm{air}}{f}^2{\mathrm{\Phi }}_m{\overline{C}}_L\overline{c}{R}^3{\widehat{r}}_2^2$$

(6)

where the detailed averaging analysis with time scale separation can be found in [13].

According to the wing kinematics modulation technique introduced by [13,33], below we demonstrate how our twin-motor driven tailless FWMAV generates lift and control torque solely through storke kinematic variations of its wings.

First, let the nominal amplitude be denoted as ${\mathrm{\Phi }}_0$, corresponding to lift $F_0={\overline{F}}_L\left({\mathrm{\Phi }}_0\right)$. Let the variation in amplitude be $\delta {\mathrm{\Phi }}_0$, as shown in Figure 3a, then the variation in lift is:

$$\begin{array}{cc}\hfill \delta F_Z& =2{\overline{F}}_L({\mathrm{\Phi }}_0+\delta {\mathrm{\Phi }}_0)-2{\overline{F}}_L\left({\mathrm{\Phi }}_0\right)={\rho }_{\mathrm{air}}{f}^2{\overline{C}}_L\overline{c}{R}^3{\widehat{r}}_2^2\delta {\mathrm{\Phi }}_0\hfill \end{array}$$

(7)

Second, as shown in Figure 3b, induce a difference in amplitude between the left and right wings to generate the roll torque:

$$\begin{array}{cc}\hfill {\tau }_X& ={\overline{F}}_L({\mathrm{\Phi }}_0+{\mathrm{\Phi }}_r)y_{cp}-{\overline{F}}_L({\mathrm{\Phi }}_0-{\mathrm{\Phi }}_r)y_{cp}={\rho }_{\mathrm{air}}{f}^2{\overline{C}}_L\overline{c}{R}^3{\widehat{r}}_2^2{y}_{cp}{\mathrm{\Phi }}_r\hfill \end{array}$$

(8)

Third, altering the center surface of the stroke motion by an angle ${\mathrm{\Phi }}_p$ to generate pitch torque, as shown in Figure 3c, considering that ${\mathrm{\Phi }}_p$ is quite small, yeilds:

$$\begin{array}{cc}\hfill {\tau }_Y& =2{\overline{F}}_L\left({\mathrm{\Phi }}_0\right)y_{cp}sin\left(\mathrm{\Phi }p\right)\approx {\rho }_{\mathrm{air}}{f}^2{\overline{C}}_L\overline{c}{R}^3{\widehat{r}}_2^2{y}_{cp}{\mathrm{\Phi }}_0\mathrm{\Phi }p\hfill \end{array}$$

(9)

Last, utilizing the split cycle parameter $\delta \sigma$ to generate the angular velocity difference between left and right wings, thereby generating yaw torque, as shown in Figure 3d. Here, we calculate ${\tau }_Z$ during the downstroke (with the upstroke following a similar process). According to the process of calculating $M_{Z^{\prime }}$ in Equation (4) and the synthesis of spatial moment vectors, after numerical approximation, the following holds:

$$\begin{array}{cc}\hfill {\tau }_Z& ={\displaystyle \frac{1}{2}}{\rho }_{\mathrm{air}}{({\displaystyle \frac{0.5}{0.5-\delta \sigma }}f)}^2{\mathrm{\Phi }}_0{\overline{C}}_D\overline{c}{R}^4{\widehat{r}}_3^3-{\displaystyle \frac{1}{2}}{\rho }_{\mathrm{air}}{({\displaystyle \frac{0.5}{0.5+\delta \sigma }}f)}^2{\mathrm{\Phi }}_0{\overline{C}}_D\overline{c}{R}^4{\widehat{r}}_3^3\hfill \\ \hfill & \approx 16{\rho }_{\mathrm{air}}{f}^2{\overline{C}}_D\overline{c}{R}^4{\widehat{r}}_3^3{\mathrm{\Phi }}_0\delta \sigma \hfill \end{array}$$

(10)

Through the aforementioned analysis, it is evident that $\delta F_Z\propto \delta {\mathrm{\Phi }}_0,{\tau }_X\propto \mathrm{\Phi }r,{\tau }_Y\propto \mathrm{\Phi }p,{\tau }_Z\propto \delta \sigma .$ Sinusoidal voltage signals are applied to the left and right motors to achieve these wing kinematics variations through four control parameters, $\mathsf{\Delta }{U}_a$, $U_r$, $U_p$, and $\mathsf{\Delta }\sigma$, corresponding to $\delta {\mathrm{\Phi }}_0$, ${\mathrm{\Phi }}_r$, ${\mathrm{\Phi }}_p$, and $\delta \sigma$ above. In addition, let $U_a=U_h+\mathsf{\Delta }{U}_a$, where $U_h$ is the voltage amplitude which generates sufficient lift force to counter the gravity. The wing flapping angle exhibit a quasi-sinusoidal waveform lagging behind the voltage signal, and these two sets of parameters are roughly proportional according to the analysis in [27]:

$${\mathrm{\Phi }}_i\propto U_i,\delta \sigma \propto \mathsf{\Delta }\sigma$$

(11)

where $i=a,rorp$. Specifically, the left and right motor input signals $u_0$ and $u_1$ at time t within one period T can be expressed as follows:

$$u_0=-\left\{\begin{array}{cc}(U_a-U_r)cos\left({\displaystyle \frac{\pi t}{(0.5-\mathsf{\Delta }\sigma )T}}\right)+U_p,\hfill & \mathrm{if}0\le t\le (0.5-\mathsf{\Delta }\sigma )T\hfill \\ (U_a-U_r)cos\left({\displaystyle \frac{\pi (T-t)}{(0.5+\mathsf{\Delta }\sigma )T}}\right)+U_p,\hfill & \mathrm{else}(0.5-\mathsf{\Delta }\sigma )T\le t\le T\hfill \end{array}\right.$$

(12)

$$u_1=\left\{\begin{array}{cc}(U_a+U_r)cos\left({\displaystyle \frac{\pi t}{(0.5+\mathsf{\Delta }\sigma )T}}\right)+U_p,\hfill & \mathrm{if}0\le t\le (0.5+\mathsf{\Delta }\sigma )T\hfill \\ (U_a+U_r)cos\left({\displaystyle \frac{\pi (T-t)}{(0.5-\mathsf{\Delta }\sigma )T}}\right)+U_p,\hfill & \mathrm{else}(0.5+\mathsf{\Delta }\sigma )T\le t\le T\hfill \end{array}\right.$$

(13)

As the bi-directional motor driven FWMAV relies solely on two motors to fully control the six degrees of freedom, it suffers from severe underactuation. Additionally, due to the high sensitivity of this type of FWMAV, even minor adjustments in wing kinematics can result in significant changes in lift and torque outputs [11], posing a considerable challenge to its control.

2.3. Reinforcement Learning

Note that the symbols in the following discussion may conflict with those used earlier, but should not cause confusion in context. See the notation section at the end for details.

Reinforcement learning has increasingly been applied to address control problems in robotics, including aerial vehicles. In typical reinforcement learning setups, an agent continually interacts with the environment: it observes the current state $s_t$, takes action $a_t$ according to its policy $\pi$, receives a reward $r_{t+1}$ from the environment, and transitions to the next state. Through these repeated iterations, the agent learns about the task and refines its policy accordingly, as shown in Figure 4a. The yellow line represents the learning process in one step. Define the return $R_t$ as the weighted sum of rewards over time:

$$R_t=\sum _{i=t}^T{\gamma }^{(i-t)}r\left(s_i,a_i\right)$$

(14)

where $\gamma$ is a discount factor between 0 and 1, used to balance the importance of current and future rewards. A $\gamma$ closer to 1 emphasizes long-term rewards, but setting it too close to 1 can hinder training performance. The key mechanism of reinforcement learning is to find a policy $\pi$ which can execute optimal actions to maximize the return $R_t$ [34].

At time t, the return $R_t$ is a random variable, but the agent needs to predict the value of $R_t$, estimating the quality of future rewards to update the policy. At this point, taking the expectation of $R_t$ can eliminate its randomness. Define the action-value function $Q^{\pi }$ as the expectation of $R_t$ with respect to $S_{t+1},A_{t+1},\cdots ,S_T,A_T$ under the condition of observing $S_t=s_t,A_t=a_t$. The $Q^{\pi }$ depends on $s_t$ and $a_t$ but does not depend on the states and actions at time t + 1 and beyond. Mathematically, it is expressed as: $Q^{\pi }\left(s_t,a_t\right)={\mathbb{E}}_{S_{i>t},A_{i>t}}\left[R_t\mid S_t=s_t,A_t=a_t\right]$. $Q^{\pi }$ can be used to measure the quality of taking action $a_t$ in state $s_t$ throughout the entire training episode when using policy $\pi$.

Define the state-value function $V^{\pi }$ as the expectation of the action-value function with respect to the action variable at the current time t: $V^{\pi }\left(s_t\right)={\mathbb{E}}_{A_t}\left[Q^{\pi }\left(s_t,A_t\right)\right]$. In policy learning, the objective function is defined as:

$$J\left(\theta \right)={\mathbb{E}}_S\left[V^{\pi }\left(S\right)\right]$$

(15)

where $\theta$ represents the parameters of the policy $\pi$. The goal is to maximize $J\left(\theta \right)$.

One common approach to policy optimization in reinforcement learning is the Actor-Critic method [35]. In this method, the Critic module approximates $Q^{\pi }(S,A)$ using the value network q with parameters $\omega$, which it provides to the Actor to improve the policy, as illustrated in Figure 4b. Then ${\nabla }_{\theta }J\left(\theta \right)$ can be approximated as $q(s,a;\omega )·{\nabla }_{\theta }ln\pi (a\mid s;\theta )$ according to the policy gradient theorem. Define the loss function of the value network using temporal-difference method:

$$L\left(\omega \right)={\displaystyle \frac{1}{2}}{\left[q\left(s_t,a_t;\omega \right)-(r_t+\gamma ·q\left(s_{t+1},a_{t+1};\omega \right))\right]}^2$$

(16)

All gradients can be obtained through backpropagation in the neural network, updating the parameters of the value network with gradient descent and the parameters of the policy network with gradient ascent becomes straightforward:

$$\begin{array}{c}\hfill \omega \leftarrow \omega -\alpha ·{\nabla }_{w}L\left(\omega \right),\theta \leftarrow \theta +\beta ·{\nabla }_{\theta }J\left(\theta \right)\end{array}$$

(17)

where $\alpha ,\beta$ are the learning rates. The final result is the policy ${\pi }_{\theta }$ after optimization.

In this paper, the Proximal Policy Optimization (PPO) algorithm [36], a member of the Actor-Critic algorithm family, is utilized for training. Unlike the original Actor-Critic algorithm, its Value Net $v_{\omega }\left(s\right)$ directly estimates $V^{\pi }$. To reduce the variance of policy gradient estimates and improve learning efficiency, the advantage function is computed using Generalized Advantage Estimation(GAE) method [37] within a given length-T trajectory segment:

$${\widehat{A}}_t=\sum _{l=0}^{T-t}{\left(\gamma \lambda \right)}^l\left(r_{t+l}+\gamma v_{\omega }\left(s_{t+l+1}\right)-v_{\omega }\left(s_{t+l}\right)\right)$$

(18)

In PPO, the trust region approach is used to approximate the objective function $J\left(\theta \right)$ of the Policy Net, and a clipping technique is typically employed to ensure that the disparity between new and old parameters remains within acceptable bounds. Relevant papers usually use $L\left(\theta \right)$ instead of $J\left(\theta \right)$ in their notation [36], which can specifically be expressed as:

$$L_t\left(\theta \right)=min\left(rati{o}_{{\pi }_{\theta }}{\widehat{A}}_t,clip\left(rati{o}_{{\pi }_{\theta }},1-ϵ,1+ϵ\right){\widehat{A}}_t\right)$$

(19)

where $rati{o}_{{\pi }_{\theta }}={\pi }_{\theta }\left(a_t\mid s_t\right)/{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\mid s_t\right)$.

3. Experiments

Based on the aforementioned physical model, a simulation environment is constructed by using the PyBullet physics engine [38]. The details of the system identification will be provided in another concurrently conducted work. This simulation is fully compatible with the OpenAI Gym environment. It can run independently on each CPU core, enabling parallel training to expedite sample collection efficiency and enhance training speed. The physics engine update frequency is set to 24 kHz, while the control step frequency is set to 1.2 kHz based on the performance of microcontrollers in reality. The steps used in reinforcement learning experiments are the control steps mentioned here. All experiments in this paper are conducted on a personal computer equipped with an i7-13790F processor and running Ubuntu 20.04.

In the context of the hovering for FWMAV, two prerequisites are paramount. Firstly, the FWMAV must be capable of generating adequate lift and three-axis torque. Secondly, it necessitates the implementation of a pose controller. In the following sections of this chapter, we discuss the relationship between the FWMAV voltage signal input and the force and torque output. Additionally, we detail the training of a pose controller using reinforcement learning.

3.1. Force and Torque

In this section, we measure the forces and torques experienced by the entire FWMAV during the flapping wing process, in preparation for subsequent reinforcement learning experiments. Additionally, an approximate linear relationship is verified between the relative control inputs $U_a$, $U_r$, $U_p$, $\mathsf{\Delta }\sigma$ and $F_Z$, ${\tau }_X$, ${\tau }_Y$, ${\tau }_Z$.

We fix the torso of the drone and measure the forces acting on the object attached to it, through which the forces exerted on the FWMAV can be inferred, akin to using force sensors in reality. Specifically, the voltages of the motors on the left and right sides are given as $u_0=-15cos2\pi ft$ and $u_1=15cos2\pi ft$. This corresponds to setting $U_a=15$ V and zeroing the four control inputs $\mathsf{\Delta }{U}_a,U_r,U_p,\mathsf{\Delta }\sigma$ specified in Equations (12) and (13). All measurements start from the FWMAV’s static state. The data for the first few wing strokes represent the transient transition state, as shown in Figure 5. Only the voltage signal and wing motion for the one side is plotted. Both the force and torque undergo median filtering with a kernel size of 3.

As shown in Figure 5a,b, though the wing stroke angle and the wing rotation angle both lag behind the voltage signal, the phase lag of the wing rotation angle is larger. In Figure 5c, the FWMAV is only subjected to the force of gravity initially, resulting in a force of approximately $-0.1$ N in the Z-axis direction. As the wings start flapping, lift force is generated. At times, negative lift occurring, this phenomenon is consistent with the findings of Dickinson [29], who attribute it to the lagging rotational motion of the wings behind the stroke motion. While in subsequent reinforcement learning experiments, asymmetric mass properties are introduced to better simulate real-world imperfections in manufacturing and assembly, the URDF mass file is configured symmetrically here for clarity. Nonetheless, some noise still persists here due to computational errors. It’s noticeable that in the absence of control, the pitch torque significantly surpasses the roll and yaw torques, as shown in Figure 5d–f. This warrants particular consideration when crafting reinforcement learning reward functions later on.

Subsequently, the mean force and torque exerted on the FWMAV during a certain time are measured with variations of control inputs. Adopting a similar force measurement method as the previous experiment and following the averaging theory, we record the data between 1 s and 5 s and plot mean values as small dots in Figure 6. In each experiment, only one of the four control inputs, $\mathsf{\Delta }{U}_a$, $U_r$, $U_p$, $\mathsf{\Delta }\sigma$, is varied. The thin red line represents the linear regression model using the least squares method. From the first row of Figure 6, it can be observed that $\delta F_z$, ${\tau }_X$, ${\tau }_Y$, and ${\tau }_Z$ exhibit proportional relationships with $\mathsf{\Delta }{U}_a$, $U_r$, $U_p$, and $\mathsf{\Delta }\sigma$, aligning with the theoretical models in Equations (7)–(11). The second row of Figure 6 illustrates how ${\tau }_X$, ${\tau }_Y$, and ${\tau }_Z$ vary with $U_a$. Notably, changes in $U_a$ have minimal impact on roll torque, while pitch torque and yaw torque demonstrate linear growth with $U_a$, consistent with Equation (8) (without ${\mathrm{\Phi }}_0$) and Equations (9) and (10) (with ${\mathrm{\Phi }}_0$), respectively.

Through the above analysis, it is evident that pitch torque is not solely controlled by $U_p$, and yaw torque is also not solely controlled by $\mathsf{\Delta }\sigma$. Both of them are influenced by $U_a$, which means the four control inputs is not decoupled. This needs to be considered in the subsequent design of the reward function. Additionally, it is observed that when $U_a$ is set to its maximum value within the operational range, 15 V, the corresponding vertical upward force is approximately $0.1$ N. For our FWMAV, the gravitational force is $-0.1095$ N, so the maximum upward acceleration is equivalent to one gravitational acceleration. Admittedly, its control authority is rather small compared to quadrotors capable of generating vertical upward accelerations exceeding three gravitational accelerations [39].

3.2. Action and State Space

In the forthcoming sections, a pose controller is developed through reinforcement learning. The discussion delve into the specifics of action and state space selection, as well as providing a description of the formulation of boundaries, rewards, and training process. In our reinforcement learning experiments, the action ${\mathit{a}}_{\mathit{t}}$ is defined as the normalized $[\mathsf{\Delta }{U}_a,U_r,U_p,\mathsf{\Delta }\sigma ]$. The state, constructed using $[{\mathit{e}}_{\mathit{P}},\mathrm{R},\mathit{v},\mathit{\omega }]\in {\mathbb{R}}^{18}$, employs a sliding window of historical observations and actions with a window size of 40 steps to address system delays and asynchronous control issues [25]. Here, ${\mathit{e}}_{\mathit{P}}=[x-x^{*},y-y^{*},z-z^{*}]$ represents the error between the current position $[x,y,z]$ and the target position $[x^{*},y^{*},z^{*}]$ in world frame. In addition, $\mathrm{R}\in {\mathbb{R}}^9$ represents the current rotation matrices. $\mathit{v}$ and $\mathit{\omega }$ denote the drone’s linear velocity and angular velocity in the body frame, respectively.

Domain randomization is adopted to enhance the robustness of the strategy and facilitate its transfer to real drones in the future. Specifically, the drone is initially placed along the Z-axis, and its altitude varies randomly between 400 mm and 600 mm, while the hover target position is always (0 mm, 0 mm, 500 mm). The Euler angles $\varphi ,\theta ,\psi$ are initially set to 0 degrees, varying randomly within $\pm 5$ degrees, with the goal of maintaining them near 0. Note that the Euler angles follow the XYZ fixed angles convention as defined in the URDF standard. Additionally, the mass properties corresponding to different wing geometries are analyzed to generate a multivariate regression function. During training, each wing’s span varies randomly within $\pm 2$ mm from a base of 90 mm and URDF file is accordingly updated, with the wing’s mass and moments of inertia automatically generated by the regression function.

3.3. Boundary and Reward

The final strategy is obtained through two sessions of training, each with different boundary conditions and reward. In the first training session, the boundaries are set as follows:

$$\left\{\begin{array}{c}\begin{array}{cc}\hfill & -100\mathrm{mm}<x<100\mathrm{mm},-100\mathrm{mm}<y<100\mathrm{mm},200\mathrm{mm}<z<800\mathrm{mm},\hfill \\ \hfill & -45\mathrm{deg}<\varphi <45\mathrm{deg},-45\mathrm{deg}<\theta <45\mathrm{deg},-45\mathrm{deg}<\psi <45\mathrm{deg}.\hfill \end{array}\hfill \end{array}\right.$$

(20)

During a training episode, the drone undergoes a reset each time it exceeds this boundary, initiating a new episode. In addition, the time boundary is set as $t<15$ s, meaning that a maximum of 18,000 control steps can be executed in one episode.

The reward function is designed to guide policy updates. Specifically, the policy, based on the previous step’s state $s_{t-1}$, applies the action $a_{t-1}$, and then the simulation computes the forward dynamics. Next, the environment calculates the reward $r_t$ using the reward function. The policy is updated based on the sequence of states, actions, and rewards. The reward function is defined as follows:

$$\begin{array}{cc}\hfill r_t=& r_p-k_p·\left|\right|{\mathit{e}}_{\mathit{p}}{\left|\right|}^2-k_a·\left(\right|\varphi |+k_{pitch}·\left|\theta \right|+\left|\psi \right|)\hfill \\ \hfill & -k_v·(1+|v_z{\left|\right)}^2-k_w·\left|\right|\mathit{\omega }\left|\right|\hfill \end{array}$$

(21)

where the term $r_p$ is initially set to 25 and increases by 5 when the drone approaches within 25 mm of the hover target point. The coefficients $k_p,k_a,k_v,k_w$ represent penalties for position error, attitude, linear velocity, and angular velocity, respectively, and are set as follows: $k_p=800$, $k_a=10$, $k_v=5$, and $k_w=0.1$. All physical quantities define in above section and can be calculated during training from the data in the state space.

Note that the linear velocity penalty term only involves $v_z$. While the hovering strategy appears to require minimizing $v_x$ and $v_y$, experimental processes indicate that directly penalizing these velocities in the reward function is ineffective. Theoretically, an ideal hovering strategy would also produce inherent oscillations in the pitch direction due to the wings’ downstroke and upstroke, which generate certain $v_x$. To address this, a coefficient $k_{pitch}$ is introduced and ultimately the trained strategy can maintain $v_x$ and $v_y$ within reasonable bounds.

The strategy trained from the first training session accomplishes hovering but exhibits notable position drift. Therefore, a second training session is carried out using the best model from the first session, along with modified boundary conditions and reward function. While other boundaries remain unchanged, position boundaries are narrowed as follows to improve precision:

$$-50\mathrm{mm}<x<50\mathrm{mm},-50\mathrm{mm}<y<50\mathrm{mm},300\mathrm{mm}<z<700\mathrm{mm},$$

(22)

and the initial value of $r_p$ in the reward function is adjusted to 50, with $k_p$ adjusted to 1600. It should be noted that without the first training session, starting from scratch with the modified conditions and parameters typically results in failure.

3.4. Training

Figure 7 illustrates the details of our parallel training framework. A separate simulation is established on each CPU core, with 20 parallel simulations used during training to enhance data collection efficiency. The sequence $s_0,a_0,r_1,s_1,a_1,r_2,\dots ,s_{n-1},a_{n-1},r_n$ in the upper right corner, is generated as follows: In each simulation environment, the state $s_0$ is constructed using the initial observation, with historical observations and actions initially set to zero. Then, the initial Policy Net generates a Gaussian distribution for the action based on $s_0$, from which the action $a_0$ is sampled. After applying $a_0$ to the FWMAV in the simulation environment, the reward $r_1$ is calculated according to the reward function. Next, state $s_1$ is constructed, and action $a_1$ is sampled from the distribution predicted by the Policy Net. After forward dynamics computation, the reward $r_2$ is obtained using the reward function. This process repeats until the sequences collected from all parallel environments reach the Rollout buffer’s size, at which point the Policy Net is updated. The black dashed box on the left contains the neural networks used in the PPO algorithm. The algorithm is implemented using the Stable-Baselines3 reinforcement learning library [40]. Both the Policy Net and Value Net are configured as neural networks with two hidden layers, each with 64 units.

In the first training session, the batch size for updating the neural network is set to 256, and the discount factor, i.e., $\gamma$ in Equation (14), is set to 0.99. In the second training session, the batch size is increased to 512, and $\lambda$ is increased to 0.995 to more thoroughly consider the expectation of future rewards. In each training session, five independent episodes are evaluated every 20,000 steps, resulting in the episode reward versus training steps curve shown as the light blue line in Figure 8. The dark blue line represents a moving average calculated every 10 data points. Subfigure (a) represents the first training session, and subfigure (b) represents the second training session.

3.5. Algorithm Comparison

To verify that the training results are not coincidental, we perform another training session from scratch using PPO with the same parameters as those used in the first session above. The reward curve obtained is shown in Figure 9. The algorithm is able to complete the hovering task, but the accuracy and response speed are certainly inferior to the policy optimized through the second session. Additionally, we train using the algorithms A2C [41] and DDPG [42], with all parameters kept the same except for the internal parameters of the algorithms. As shown in Figure 9, both algorithms fail to achieve high rewards.

4. Results

The trained policy successfully achieves the goal of hovering within a 50 mm diameter range around the initial center of mass. It is noteworthy that, although the initial and target positions are close during training, the strategy demonstrates the capability to generalize to more distant targets. The strategy enables the FWMAV to hover at any set point in world frame and perform vertical, forward and lateral flights, enabling point-to-point flight from the initial position to distant target points, which serves effectively as a low-level pose controller. As a demonstration of its versatility, we showcase trajectory tracking along a circular path and the control of multiple drones within the simulation environment.

4.1. Open-Loop Flight

As a comparison, we present open-loop flight segments where the four control inputs $\mathsf{\Delta }{U}_a,U_r,U_p,\mathsf{\Delta }\sigma$ are set to 0, and $U_a$ is set to constant values. Figure 10a shows the trajectories of the FWMAV starting from (0 mm, 0 mm, 1000 mm) for 0.8 s under different $U_a$ values. The trajectory with $U_a=9$ V is shown in red, and the trajectory with $U_a=12$ V is shown in blue. Figure 10b compares the positions and attitudes under the two $U_a$ values. It can be seen that the drone becomes unstable in a short period of time in both cases. Additionally, although $U_a=9$ V demonstrates a positive force along the z-axis in the force measurement experiment, as shown in the first subplot of Figure 6, this $U_a$ is not sufficient to lift the drone due to pitch oscillations.

4.2. Hovering

The trained policy is applied to control the FWMAV for setpoint hovering. As shown in Figure 11a, after 2 s of adjustment, the position error between the torso’s center of mass and the set point does not exceed 10 mm within the third second. Figure 11b illustrates the position and attitude of the FWMAV hovering within 20 s. Note that the FWMAV exhibits inherent oscillations in the pitch direction due to wing flapping, leading to position fluctuations in the x direction. Since the state space during training observed the error between the current position and the target position, our policy can achieve hovering at any arbitrary point in space. By performing a coordinate transformation before inputting the state into the policy, the FWMAV can hover with different yaw angles. These two points are demonstrated in the experimental results of the following sections.

4.3. Point to Point Flight

The trained policy enables the drone to fly to set points farther than those encountered during training and maintain hovering at these positions. Figure 12a demonstrates the FWMAV lifting off from (0 mm, 0 mm, 500 mm) and hovering at (0 mm, 0 mm, 1000 mm). Figure 12b illustrates the FWMAV flying forward from (0 mm, 0 mm, 500 mm) and hovering at (500 mm, 0 mm, 500 mm). Figure 12c shows the FWMAV flying laterally from (0 mm, 0 mm, 500 mm) and hovering at (0 mm, 500 mm, 500 mm). As shown in the three sets of images mentioned above, during the execution of these three tasks, except for the initial 1 s, the pitch angles all remain within ±5 degrees, and the roll and yaw angles all remain within ±2 degrees. Figure 12d depicts the FWMAV taking off at a yaw angle of 45 degrees from (0 mm, 0 mm, 200 mm) and hovering at (500 mm, 500 mm, 700 mm), where the pitch and roll angles converge to a smaller range after oscillation.

It is worth noting that directly training for these long-distance point-to-point flights is rather difficult. The drone must learn to accelerate for takeoff and then decelerate to hover. If the starting and target positions are set as far apart as in this section during training, it often results in failures. However, this kind of trianing task is relatively easy for quadcopters with greater control authority [43].

4.4. Trajectory Tracking

The trained policy can be used for trajectory tracking. Figure 13 illustrates the drone tracking a circle parallel to the horizontal plane ith a radius of 300 mm. The target position setpoint is changed every 600 control steps and it is set to a point tangent to the trajectory in order to accelerate the tracking speed.

4.5. Multi-Drones

This section demonstrates implementation of controlled flight of multiple drones in a simulated environment. Figure 14a shows multiple drones hovering at different vertex positions of a 300 mm edge length cube in space. Figure 14b presents two images showing eight drones taking off from a height of 200 mm to hover at 750 mm, each with a different initial yaw angle, while maintaining a circular formation during the flight.

During training, the FWMAV always takes off from a stationary state, but our hovering policy can be applied to drones with non-zero initial velocity. To demonstrate this, eight FWMAVs are initially positioned at the same height of 750 mm, and the 0th FWMAV is applied the trained control voltage signal, while the 1st to 7th FWMAVs are allowed to free fall under gravity. The control voltage is applied to the 1st FWMAV after the 0th FWMAV has been controlled for 20 control steps, then to the 2nd FWMAV after the 1st FWMAV has been controlled for 20 control steps, and so on. As shown in Figure 15, all eight drones eventually return to hovering at the height of 750 mm.

5. Conclusions

This paper utilizes deep reinforcement learning to train a hovering strategy for a Bi-directional Motor Driven FWMAV. The strategy operates at a low-level, directly mapping the FWMAV’s state to motor voltage outputs. We select appropriate action and state spaces, design a reward function to maintain attitude stability during flapping flight within a confined space, and train using the PPO algorithm across 20 parallel simulation environments. Verification in high-fidelity simulation environments demonstrates its applicability for FWMAV pose control, enabling agile transitions from forward flight, lateral flight, or liftoff to hovering, facilitating point-to-point movement. Additionally, simple trajectory tracking and multi-drone formation flight are demonstrated. Future work will focus on sim-to-real transfer, particularly addressing sensor noise and safety concerns, with the aim of deploying the strategy onto real FWMAVs in laboratory.

6. Notation

Because different fields have their own conventional symbols, which may conflict and cause confusion in this manuscript, we include this section to clarify the potentially confusing symbols. Below are the notation in different contexts:

r	Aerodynamics: Radial wing distance
	Reinforcement Learning: Rewards (typically with a time-series subscript)
R	Aerodynamics: Wing span
	Reinforcement Learning: Return
$\varphi$, $\psi$	Drone Euler Angles: Roll angle ($\varphi$) and yaw angle ($\psi$)
	Wing Motion: Stroke angle and rotation angle (typically with subscript w)
$\theta$	Drone Euler Angles: Pitch angle
	Policy Learning: Parameters to be optimized in the policy $\pi$
$\omega$	Drone State: Body angular velocity
	Actor-Critic Algorithm: Parameters of the value network