# Neuroevolutionary Control for Autonomous Soaring

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Soaring Problem

#### 2.1. Trajectory Optimization

**x**$\left(t\right)$, the control

**u**$\left(t\right)$, the initial time ${t}_{0}$, and the final time ${t}_{f}$ that minimizes the cost function J, with boundary conditions $\mathsf{\Phi}$ and Lagrange term performance index $\mathcal{L}$, all summarized as [21]:

#### 2.2. Flight Dynamics Model

#### 2.3. Dynamic Soaring

#### 2.4. Thermal Soaring

## 3. Neurocontrol

#### 3.1. Neural Network Topology

#### 3.2. Neuroevolutionary Strategy

#### 3.3. Neuroevolutionary Implementation

#### 3.4. Simulation Environment

Algorithm 1 Flight simulation. | |

1: for neural network ${\pi}_{N}$ in population $\Pi $ do | |

2: while $t<{t}_{f}$ do | |

3: states ←get_states() | ▹ fetch aircraft states |

4: if $states>constraints$ then | |

5: break | |

6: actions ←${\pi}_{N}$(states) | ▹ obtain control commands |

7: ${W}_{x}$, ${W}_{y}$, ${W}_{z}$←get_wind() | ▹ calculate local wind profile |

8: ${x}_{k+1}$←${x}_{k}$ + $\delta $$\dot{{x}_{k}}$ | ▹ apply dynamics model |

9: $t+=\delta $ | |

10: ${f}_{\pi}$←get_fitness() | ▹ compute rewards and penalties |

## 4. Results and Discussion

#### 4.1. Albatross Dynamic Soaring

#### 4.2. SUAV Dynamic Soaring

#### 4.3. SUAV Thermal Soaring

## 5. Conclusions

## Supplementary Materials

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Sachs, G. Minimum shear wind strength required for dynamic soaring of albatrosses. Ibis
**2005**, 147, 1–10. [Google Scholar] [CrossRef] - Pennycuick, C.J. Thermal Soaring Compared in Three Dissimilar Tropical Bird Species, Fregata Magnificens, Pelecanus Occidentals and Coragyps Atratus. J. Exp. Biol.
**1983**, 102, 307–325. [Google Scholar] [CrossRef] - Sachs, G.; Traugott, J.; Nesterova, A.P.; Dell’Omo, G.; Kümmeth, F.; Heidrich, W.; Vyssotski, A.L.; Bonadonna, F. Flying at No Mechanical Energy Cost: Disclosing the Secret of Wandering Albatrosses. PLoS ONE
**2012**, 7, e41449. [Google Scholar] [CrossRef] [PubMed] - Allen, M. Autonomous Soaring for Improved Endurance of a Small Uninhabitated Air Vehicle. In Proceedings of the 43rd AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV, USA, 10–13 January 2005; American Institute of Aeronautics and Astronautics: Reno, NV, USA, 2005. [Google Scholar] [CrossRef]
- Zhao, Y.J. Optimal patterns of glider dynamic soaring. Optim. Control Appl. Methods
**2004**, 25, 67–89. [Google Scholar] [CrossRef] - Sachs, G.; Grüter, B. Trajectory Optimization and Analytic Solutions for High-Speed Dynamic Soaring. Aerospace
**2020**, 7, 47. [Google Scholar] [CrossRef] [Green Version] - Shaw-Cortez, W.E.; Frew, E. Efficient Trajectory Development for Small Unmanned Aircraft Dynamic Soaring Applications. J. Guid. Control Dyn.
**2015**, 38, 519–523. [Google Scholar] [CrossRef] - Akhtar, N.; Whidborne, J.F.; Cooke, A.K. Real-time optimal techniques for unmanned air vehicles fuel saving. Proc. Inst. Mech. Eng. Part G J. Aerosp. Eng.
**2012**, 226, 1315–1328. [Google Scholar] [CrossRef] - Gao, C.; Liu, H.H.T. Dubins path-based dynamic soaring trajectory planning and tracking control in a gradient wind field. Optim. Control Appl. Methods
**2017**, 38, 147–166. [Google Scholar] [CrossRef] - Lawrance, N.R.J.; Sukkarieh, S. A guidance and control strategy for dynamic soaring with a gliding UAV. In Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan, 12–17 May 2009; pp. 3632–3637. [Google Scholar] [CrossRef]
- Pogorzelski, G.; Silvestre, F.J. Autonomous soaring using a simplified MPC approach. Aeronaut. J.
**2019**, 123, 1666–1700. [Google Scholar] [CrossRef] - Kim, S.h.; Lee, J.; Jung, S.; Lee, H.; Kim, Y. Deep Neural Network-Based Feedback Control for Dynamic Soaring of Unpowered Aircraft. IFAC-PapersOnLine
**2019**, 52, 117–121. [Google Scholar] [CrossRef] - Montella, C.; Spletzer, J.R. Reinforcement learning for autonomous dynamic soaring in shear winds. In Proceedings of the 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA, 14–18 September 2014; pp. 3423–3428. [Google Scholar] [CrossRef]
- Woodbury, T.D.; Dunn, C.; Valasek, J. Autonomous Soaring Using Reinforcement Learning for Trajectory Generation. In 52nd Aerospace Sciences Meeting; AIAA SciTech Forum; American Institute of Aeronautics and Astronautics: National Harbor, MD, USA, 2014. [Google Scholar] [CrossRef] [Green Version]
- Chung, J.; Lawrance, N.; Sukkarieh, S. Learning to soar: Resource-constrained exploration in reinforcement learning. Int. J. Robot. Res.
**2014**, 34, 158–172. [Google Scholar] [CrossRef] - Muliadi, J.; Kusumoputro, B. Neural Network Control System of UAV Altitude Dynamics and Its Comparison with the PID Control System. J. Adv. Transp.
**2018**, 2018, 3823201. [Google Scholar] [CrossRef] - Efe, M.Ö. Neural Network Assisted Computationally Simple PID Control of a Quadrotor UAV. IEEE Trans. Ind. Inform.
**2011**, 7, 354–361. [Google Scholar] [CrossRef] - Bansal, S.; Akametalu, A.K.; Jiang, F.J.; Laine, F.; Tomlin, C.J. Learning Quadrotor Dynamics Using Neural Network for Flight Control. arXiv
**2016**, arXiv:1610.05863. [Google Scholar] - Stanley, K.O.; Miikkulainen, R. Evolving Neural Networks through Augmenting Topologies. Evol. Comput.
**2002**, 10, 99–127. [Google Scholar] [CrossRef] [PubMed] - Perez, R.E.; Arnal, J.; Jansen, P.W. Neuro-Evolutionary Control for Optimal Dynamic Soaring. In Proceedings of the AIAA Scitech 2020 Forum, Orlando, FL, USA, 6–10 January 2020; American Institute of Aeronautics and Astronautics: Orlando, FL, USA, 2020. [Google Scholar] [CrossRef]
- Rao, A. A Survey of Numerical Methods for Optimal Control. Adv. Astronaut. Sci.
**2010**, 135, 497–528. [Google Scholar] - Betts, J.T. Survey of Numerical Methods for Trajectory Optimization. J. Guid. Control Dyn.
**1998**, 21, 193–207. [Google Scholar] [CrossRef] - Miele, A. Flight Mechanics, Theory of Flight Paths; Addison-Wesley: Reading, MA, USA, 1962; Volume 1. [Google Scholar]
- Lawrance, N.; Sukkarieh, S. Wind Energy Based Path Planning for a Small Gliding Unmanned Aerial Vehicle. In Proceedings of the AIAA Guidance, Navigation, and Control Conference, Chicago, IL, USA, 10–13 August 2009; American Institute of Aeronautics and Astronautics: Chicago, IL, USA, 2009. [Google Scholar] [CrossRef]
- Perez, R.E.; Jansen, P.W.; Martins, J.R.R.A. pyOpt: A Python-based object-oriented framework for nonlinear constrained optimization. Struct. Multidiscip. Optim.
**2012**, 45, 101–118. [Google Scholar] [CrossRef] - Li, Z.; Langelaan, J.W. Parameterized Trajectory Planning for Dynamic Soaring. In Proceedings of the AIAA Scitech 2020 Forum, Orlando, FL, USA, 6–10 January 2020; American Institute of Aeronautics and Astronautics: Orlando, FL, USA, 2020. [Google Scholar] [CrossRef]

**Figure 2.**Typical dynamic soaring trajectory with a linear wind gradient (adapted from [5]).

**Figure 3.**Horizontal wind profile shapes, with ${W}_{{max}_{x}}=10\phantom{\rule{4.pt}{0ex}}\mathrm{m}/\mathrm{s}$ and ${h}_{t{r}_{x}}=30\phantom{\rule{4.pt}{0ex}}\mathrm{m}$.

**Figure 8.**Simulated trajectories of the dynamic soaring albatross neurocontroller, with start marker (green), end marker (red), and path (blue).

**Figure 16.**Aircraft visualization of the dynamic soaring SUAV neurocontroller (Supplementary Materials). The attitude indicator displays the roll angle along its circumference, as well as the pitch angle through the concentric circles within the plot. The shaded red zones represent the roll angle limits. The heading and wind velocities are shown by the blue and green vectors, respectively, where their orientations indicate direction and their lengths indicate the speed.

**Figure 17.**NEAT and TO simulated trajectories of the dynamic soaring SUAV. Neurocontroller in solid, optimized trajectory in dotted.

**Figure 18.**NEAT and TO traces of the dynamic soaring SUAV. The neurocontroller in solid lines; the optimized trajectory in dotted lines.

Albatross Parameters | Value | Wind Parameters | Value | Trajectory Parameters | Value |
---|---|---|---|---|---|

g (m/${\mathrm{s}}^{2}$) | 9.8 | ${A}_{x}$ | 1.0 | ${V}_{0}$ (m/${\mathrm{s}}^{2}$) | 9.1 |

$\rho $ (kg/${\mathrm{m}}^{3}$) | 1.225 | ${h}_{t{r}_{x}}$ (m) | 9.1 | ${\psi}_{0}$ (deg) | −25 |

m (kg) | 8.5 | ${W}_{{max}_{x}}$ (m/${\mathrm{s}}^{2}$) | 10.2 | ${\gamma}_{0}$ (deg) | 0 |

S (${\mathrm{m}}^{2}$) | 0.65 | ${h}_{0}$ (m) | 6.1 | ||

${C}_{{D}_{0}}$ | 0.033 | ${A}_{y}$ | 1.0 | ${x}_{0}$ (m) | 0 |

${E}_{max}$ | 20.0 | ${h}_{t{r}_{y}}$ (m) | 9.1 | ${y}_{0}$ (m) | 0 |

${n}_{max}$ | 5 | ${W}_{{max}_{y}}$ (m/${\mathrm{s}}^{2}$) | 4.8 | ${h}_{min}$ (m) | 0 |

${C}_{{L}_{max}}$ | 1.6 | ${\dot{\gamma}}_{max}$ (deg/s) | 100 | ||

${C}_{{L}_{min}}$ | −0.25 | ${\dot{\psi}}_{max}$ (deg/s) | 100 | ||

${\mu}_{max}$ (deg) | 60 | ${\dot{{C}_{L}}}_{max}$ | 0.25 | ||

${\dot{\mu}}_{max}$ (deg/s) | 90 | ||||

${t}_{f}$ (s) | 600 |

SUAV Parameters | Value | Wind Parameters | Value | Trajectory Parameters | Value |
---|---|---|---|---|---|

g (m/${\mathrm{s}}^{2}$) | 9.8 | ${A}_{x}$ | 1.0 | ${V}_{0}$ (m/${\mathrm{s}}^{2}$) | 10.7 |

$\rho $ (kg/${\mathrm{m}}^{3}$) | 1.225 | ${h}_{t{r}_{x}}$ (m) | 304.8 | ${\psi}_{0}$ (deg) | 0 |

m (kg) | 4.3 | ${W}_{{max}_{x}}$ (m/${\mathrm{s}}^{2}$) | 6.1 | ${\gamma}_{0}$ (deg) | 0 |

S (${\mathrm{m}}^{2}$) | 1.0 | ${h}_{0}$ (m) | 301.8 | ||

${C}_{{D}_{0}}$ | 0.025 | ${A}_{y}$ | N/A | ${x}_{0}$ (m) | 0 |

${E}_{max}$ | 20.0 | ${h}_{t{r}_{y}}$ (m) | N/A | ${y}_{0}$ (m) | 0 |

${n}_{max}$ | 15 | ${W}_{{max}_{x}}$ (m/${\mathrm{s}}^{2}$) | N/A | ${h}_{min}$ (m) | 0 |

${C}_{{L}_{max}}$ | 1.5 | ${\dot{\gamma}}_{max}$ (deg/s) | 100 | ||

${C}_{{L}_{min}}$ | −0.2 | ${\dot{\psi}}_{max}$ (deg/s) | 100 | ||

${\mu}_{max}$ (deg) | 60 | ${\dot{{C}_{L}}}_{max}$ | 0.25 | ||

${\dot{\mu}}_{max}$ (deg/s) | 90 | ||||

${t}_{f}$ (s) | 600 |

Boundary ${\mathit{t}}_{0}$ | Value | Boundary ${\mathit{t}}_{\mathit{f}}$ | Value | Path | Min | Max |
---|---|---|---|---|---|---|

${V}_{0}$ (m/s) | 12.5 | ${V}_{f}$ (m/s) | 12.5 | V (m/s) | 0.0 | 1000.0 |

${\psi}_{0}$ (deg) | 60.0 | ${\psi}_{f}$ (deg) | 60.0 | $\dot{\psi}$ (deg/s) | −100 | 100 |

${\gamma}_{0}$ (deg) | 0.0 | ${\gamma}_{f}$ (deg) | 0.0 | $\dot{\gamma}$ (deg/s) | −100 | 100 |

${z}_{0}$ (m) | 300.0 | ${z}_{f}$ (m) | 300.0 | n | 0.0 | 15.0 |

${x}_{0}$ (m) | 0.0 | ${C}_{L}$ | −0.2 | 1.5 | ||

${y}_{0}$ (m) | 0.0 | |||||

${C}_{{L}_{0}}$ | 1.0 | ${C}_{{L}_{f}}$ | 1.0 | |||

${\mu}_{0}$ (deg) | −40.0 | ${\mu}_{f}$ (deg) | −10.0 |

SUAV Parameters | Value | Wind Parameters | Value | Trajectory Parameters | Value |
---|---|---|---|---|---|

g (m/${\mathrm{s}}^{2}$) | 9.8 | ${W}_{core}$ (m/s) | 3.05 | ${V}_{0}$ (m/${\mathrm{s}}^{2}$) | 9.1 |

$\rho $ (kg/${\mathrm{m}}^{3}$) | 1.225 | ${h}_{{t}_{0}}$ (m) | 91.4 | ${\psi}_{0}$ (deg) | 0 |

m (kg) | 4.3 | ${r}_{xy}$ (m) | 30.5 | ${\gamma}_{0}$ (deg) | 0 |

S (${\mathrm{m}}^{2}$) | 1.0 | ${r}_{z}$ (m) | 61.0 | ${h}_{0}$ (m) | 106.7 |

${C}_{{D}_{0}}$ | 0.025 | $\dot{{h}_{t}}$ (m/s) | 0.213 | ${x}_{0}$ (m) | 0 |

${E}_{max}$ | 20.0 | ${y}_{0}$ (m) | −38.1 | ||

${n}_{max}$ | 15 | ${W}_{{max}_{x}}$ (m/${\mathrm{s}}^{2}$) | N/A | ${h}_{min}$ (m) | 0 |

${C}_{{L}_{max}}$ | 1.5 | ${W}_{{max}_{x}}$ (m/${\mathrm{s}}^{2}$) | N/A | ${\dot{\gamma}}_{max}$ (deg/s) | 100 |

${C}_{{L}_{min}}$ | −0.2 | ${\dot{\psi}}_{max}$ (deg/s) | 10 | ||

${\mu}_{max}$ (deg) | 60 | ${\dot{{C}_{L}}}_{max}$ | 0.25 | ||

${\dot{\mu}}_{max}$ (deg/s) | 90 | ||||

${t}_{f}$ (s) | 600 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kim, E.J.; Perez, R.E.
Neuroevolutionary Control for Autonomous Soaring. *Aerospace* **2021**, *8*, 267.
https://doi.org/10.3390/aerospace8090267

**AMA Style**

Kim EJ, Perez RE.
Neuroevolutionary Control for Autonomous Soaring. *Aerospace*. 2021; 8(9):267.
https://doi.org/10.3390/aerospace8090267

**Chicago/Turabian Style**

Kim, Eric J., and Ruben E. Perez.
2021. "Neuroevolutionary Control for Autonomous Soaring" *Aerospace* 8, no. 9: 267.
https://doi.org/10.3390/aerospace8090267