# Model-Based Tracking of Fruit Flies in Free Flight

^{1}

^{2}

^{3}

^{*}

^{†}

## Abstract

**:**

## Simple Summary

## Abstract

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Problem Definition

#### 2.2. Experimental Setup

#### 2.3. Background Subtraction

#### 2.4. Generative Model

#### 2.5. Loss Function Optimization

## 3. Results

#### 3.1. Validation

#### 3.2. Non-Maneuvering Flight

#### 3.3. Free Flight Maneuver

#### 3.4. Roll Correction Maneuver

## 4. Discussion and Conclusions

## Supplementary Materials

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

DOF | degree of freedom |

## Appendix A. Comparison to 3-Camera Hull Reconstruction

**Figure A1.**Comparing the 17-DOF model to hull reconstruction tracking in a Cartesian camera configuration. The six plots show 25 ms of wing angles data for both wings, comparing the results of two algorithms: The 17-DOF fitted model reported here (blue) and the hull reconstruction tracking reported in [11,13,14] (red). The video was taken at 8000 frames per second and $512\phantom{\rule{3.33333pt}{0ex}}\times \phantom{\rule{3.33333pt}{0ex}}512$ pixel resolution.

**Figure A2.**Comparing the 17-DOF model to hull reconstruction tracking in the current camera configuration. The unperturbed flight video from Figure 9 was analyzed using a hull reconstruction algorithm which approximates the performance of the methods reported in [13,14] by adjusting the latter method to the current camera configuration, higher resolution and higher frame rate. (

**a**) The elevation angle of the right wing ${\theta}_{R}$ as a function of the stroke angle of the same wing ${\varphi}_{R}$. Results of the 17-DOF model (red) are less noisy and more self-consistent than the results of the hull reconstruction method, especially when the wings are at the back (${\varphi}_{R}=130-{180}^{\xb0}$). (

**b**) The reconstructed hull and the fitted 17-DOF model, both superimposed. In this video frame, when the wings are at the back, wing–wing and body–wing occlusions result in a oversized wing hull, which causes errors in the wing pose estimation. For example, both chord vectors are wrongly identified by the hull reconstruction method (red and blue line). In this example, the wings’ DOFs are correctly identified by the 17-DOF model.

## References

- Dudley, R. The Biomechanics of Insect Flight: Form, Function, Evolution; Princeton University Press: Princeton, NJ, USA, 2002. [Google Scholar]
- Dickinson, M.H.; Muijres, F.T. The aerodynamics and control of free flight manoeuvres in Drosophila. Phil. Trans. R. Soc. B
**2016**, 371, 20150388. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Floreano, D.; Wood, R.J. Science, technology and the future of small autonomous drones. Nature
**2015**, 521, 460. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Fry, S.N.; Sayaman, R.; Dickinson, M.H. The aerodynamics of hovering flight in Drosophila. J. Exp. Biol.
**2005**, 208, 2303–2318. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Kassner, Z.; Dafni, E.; Ribak, G. Kinematic compensation for wing loss in flying damselflies. J. Insect Physiol.
**2016**, 85, 1–9. [Google Scholar] [CrossRef] - Lyu, Y.Z.; Zhu, H.J.; Sun, M. Wing kinematic and aerodynamic compensations for unilateral wing damage in a small phorid fly. Phys. Rev. E
**2020**, 101, 012412. [Google Scholar] [CrossRef] - Hedrick, T.L. Software techniques for two- and three-dimensional kinematic measurements of biological and biomimetic systems. Bioinspir. Biomim.
**2008**, 3, 034001. [Google Scholar] [CrossRef] - Koehler, C.; Liang, Z.; Gaston, Z.; Wan, H.; Dong, H. 3D reconstruction and analysis of wing deformation in free-flying dragonflies. J. Exp. Biol.
**2012**, 215, 3018. [Google Scholar] [CrossRef] [Green Version] - Nasir, N.; Mat, S. An automated visual tracking measurement for quantifying wing and body motion of free-flying houseflies. Measurement
**2019**, 143, 267–275. [Google Scholar] [CrossRef] - Wang, H.; Zeng, L.; Yin, C. Measuring the body position, attitude and wing deformation of a free-flight dragonfly by combining a comb fringe pattern with sign points on the wing. Meas. Sci. Technol.
**2002**, 13, 903–908. [Google Scholar] [CrossRef] - Ristroph, L.; Berman, G.J.; Bergou, A.J.; Wang, Z.J.; Cohen, I. Automated hull reconstruction motion tracking (HRMT) applied to sideways maneuvers of free-flying insects. J. Exp. Biol.
**2009**, 212, 1324–1335. [Google Scholar] [CrossRef] - Walker, S.M.; Thomas, A.L.R.; Taylor, G.K. Operation of the alula as an indicator of gear change in hoverflies. J. R. Soc. Interface
**2012**, 9, 1194–1207. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Beatus, T.; Guckenheimer, J.; Cohen, I. Controlling roll perturbations in fruit flies. J. Roy. Soc. Interface
**2015**, 12, 20150075. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Whitehead, S.C.; Beatus, T.; Canale, L.; Cohen, I. Pitch perfect: How fruit flies control their body pitch angle. J. Exp. Biol.
**2015**, 218, 3508–3519. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Bomphrey, R.J.; Nakata, T.; Phillips, N.; Walker, S.M. Smart wing rotation and trailing-edge vortices enable high frequency mosquito flight. Nature
**2017**, 544, 92–95. [Google Scholar] [CrossRef] - Fontaine, E.I.; Zabala, F.; Dickinson, M.H.; Burdick, J.W. Wing and body motion during flight initiation in Drosophila revealed by automated visual tracking. J. Exp. Biol.
**2009**, 212, 1307–1323. [Google Scholar] [CrossRef] [Green Version] - Muijres, F.T.; Chang, S.W.; van Veen, W.G.; Spitzen, J.; Biemans, B.T.; Koehl, M.A.R.; Dudley, R. Escaping blood-fed malaria mosquitoes minimize tactile detection without compromising on take-off speed. J. Exp. Biol.
**2017**, 220, 3751–3762. [Google Scholar] [CrossRef] [Green Version] - Kostreski, N.I. Automated Kinematic Extraction of Wing and Body Motions of Free Flying Diptera. Ph.D. Thesis, University of Maryland, College Park, MD, USA, 2012. [Google Scholar]
- Muijres, F.T.; Elzinga, M.J.; Melis, J.M.; Dickinson, M.H. Flies evade looming targets by executing rapid visually directed banked turns. Science
**2014**, 344, 172–177. [Google Scholar] [CrossRef] - Wehmann, H.N.; Heepe, L.; Gorb, S.N.; Engels, T.; Lehmann, F.O. Local deformation and stiffness distribution in fly wings. Biol. Open
**2019**, 8, bio038299. [Google Scholar] [CrossRef] [Green Version] - Theriault, D.H.; Fuller, N.W.; Jackson, B.E.; Bluhm, E.; Evangelista, D.; Wu, Z.; Betke, M.; Hedrick, T.L. A protocol and calibration method for accurate multi-camera field videography. J. Exp. Biol.
**2014**, 217, 1843–1848. [Google Scholar] [CrossRef] [Green Version] - Ristroph, L.; Bergou, A.J.; Ristroph, G.; Coumes, K.; Berman, G.J.; Guckenheimer, J.; Wang, Z.J.; Cohen, I. Discovering the flight autostabilizer of fruit flies by inducing aerial stumbles. Proc. Natl. Acad. Sci. USA
**2010**, 107, 4820–4824. [Google Scholar] [CrossRef] - Otsu, N. A Threshold Selection Method from Gray-Level Histograms. IEEE Trans. Syst. Man Cybern.
**1979**, 9, 62–66. [Google Scholar] [CrossRef] [Green Version] - Beatus, T.; Cohen, I. Wing-pitch modulation in maneuvering fruit flies is explained by an interplay between aerodynamics and a torsional spring. Phys. Rev. E
**2015**, 92, 022712. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Nagesh, I.; Walker, S.M.; Taylor, G.K. Motor output and control input in flapping flight: A compact model of the deforming wing kinematics of manoeuvring hoverflies. J. R. Soc. Interface
**2019**, 16, 20190435. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Combes, S.; Daniel, T. Flexural stiffness in insect wings I. Scaling and the influence of wing venation. J. Exp. Biol.
**2003**, 206, 2979–2987. [Google Scholar] [CrossRef] [Green Version] - Ma, Y.; Ma, T.; Ning, J.; Gorb, S. Structure and tensile properties of the forewing costal vein of the honeybee Apis mellifera. Soft Matter
**2020**, 16, 4057–4064. [Google Scholar] [CrossRef] - Young, J.; Walker, S.M.; Bomphrey, R.J.; Taylor, G.K.; Thomas, A.L. Details of insect wing design and deformation enhance aerodynamic function and flight efficiency. Science
**2009**, 325, 1549–1552. [Google Scholar] [CrossRef] - Arthur, D.; Vassilvitskii, S. k-means++: The advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’07), New Orleans, LA, USA, 7–9 January 2007; Society for Industrial and Applied Mathematics: Philadelphia, PA, USA, 2007; pp. 1027–1035. [Google Scholar]
- Liu, S.; Li, T.; Chen, W.; Li, H. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 7708–7717. [Google Scholar]

**Figure 1.**Basic 12-DOF model parameters. (

**a**) Body 6 DOF describing its position and orientation. (

**b**) Each wing is described by 3 Euler angles relative to the stroke plane: Stroke ($\varphi $), elevation ($\theta $) and wing pitch ($\psi $). The annotations are for the left wing.

**Figure 2.**Experimental setup. Three orthogonal high-speed cameras focused on a transparent chamber. The non-Cartesian setup reduces wing occlusions.

**Figure 3.**Pre-processing of free flying videos. (

**a**) Superimposition of 6 raw images mid flight. (

**b**) Background extracted by taking pixel-wise maximum. (

**c**) Subtraction of a single frame from the background. (

**d**) Histogram of subtracted image. The distribution is almost uni-modal. (

**e**) Histogram of subtracted image after power transformation. Distribution is more bi-modal. (

**f**) Resulting mask after applying Otsu’s threshold on transformed difference image.

**Figure 4.**Wing deformations. (

**a**) Top: Three frames from different phases of a single wing beat. Bottom: Superimposing the 3 frames shows that the wing hinge is effectively not fixed during the stroke. The solid lines marking the leading edge of the wing do not intersect at a single point (dashed lines). (

**b**) An unsuccessful fitting attempt using a rigid wing on a frame with a twisted wing during supination. (

**c**) Wing deformation used in our 3D model. Color represents deformation level, and the black line shows the rigid wing outline. (

**d**) A successful fit using a flexible wing.

**Figure 5.**Single-frame loss function. XOR operation on the camera image mask (yellow) and the projected model.

**Figure 6.**Degeneracy in $\psi $. (

**a**,

**b**) The 3D model generated by two sets of parameters. The orange arrow shows the direction of the camera taking the images on the bottom. (

**c**,

**d**) The projection of the corresponding models on the camera plane. The projections are nearly identical. (

**e**) The loss function (z-axis) at the presented frame by changing only ${\psi}_{\ell}$ and ${\varphi}_{\ell}$ (x-axis and y-axis, respectively).

**Figure 7.**Model validation on synthetic data. Tracking errors box plot. Each box contains 75% of the data. Whiskers correspond to 99.3% of the data.

**Figure 8.**Naive errors vs. error detection. Each graph is a histogram of the errors for each DOF. The blue bars are the naive optimization process and the orange bars are the process using error detection.

**Figure 9.**Results on an unperturbed flight event. (

**a**) The projection of a fitted 3D model superimposed on the corresponding frames. (

**b**) Body pitch and wing $\varphi $. Body pitch oscillations are marked in black vertical arrows. (

**c**) The path of the wing tip by its elevation ($\theta $) and azimuth ($\varphi $).

**Figure 10.**Comparison between the 12-DOF and 17-DOF models. The body roll (

**a**) and yaw (

**b**) angles found using the 12-DOF (blue) and 17-DOF (red) models, both for the same unperturbed, non-maneuvering flight data.

**Figure 11.**Free flight maneuver. (

**a**) Body angles. (

**b**) Wings stroke angles. (

**c**) A drawing of the path and body angles of the fly. The red rods represent the orientation of body at different time points, with the circular end marking the head. The blue rods attached to the red rods represent the wing span vector, visualizing the yaw and roll angles. A small circle on the left marks the start of the video.

**Figure 12.**Roll correction. (

**a**) Body angles during the maneuver. Magnetic pulse was activated between $t\phantom{\rule{3.33333pt}{0ex}}=\phantom{\rule{3.33333pt}{0ex}}0\phantom{\rule{3.33333pt}{0ex}}-\phantom{\rule{3.33333pt}{0ex}}7.5\phantom{\rule{3.33333pt}{0ex}}\mathrm{ms}$. An orange vertical arrow marks the time of maximum angular deflection in roll, yaw and pitch. (

**b**) Wings stroke angles. Blue line and red dashed line mark ${\varphi}_{\ell}$ and ${\varphi}_{r}$ respectively. The rectangle marks the main wing asymmetry during the maneuver. (

**c**) Top view of the fitted model shows every two wing beats when the left wing is at supination. Wing stroke asymmetry is clearly visible. (

**d**) Flight trajectory and body pose plotted in 1 wingbeat intervals during the maneuver. The body x axis is plotted in red lines, with a red dot indicating the head, and the body y direction is represented by blue line. The trajectory starts at the top corner of the plotted box. A 2D projection of the trajectory is plotted on the $xy$ plane (black line), with the time of the perturbation marked by a thicker line.

Name | Units | Description |
---|---|---|

$x,y,z$ | mm | Center of mass position in the lab frame |

yaw | deg | Body azimuth angle (rotation around lab z) |

pitch | deg | Body elevation angle |

roll | deg | Body rotation around body x axis |

${\varphi}_{\ell},{\varphi}_{r}$ | deg | Wing stoke angles |

${\theta}_{\ell},{\theta}_{r}$ | deg | Wing elevation angles |

${\psi}_{\ell},{\psi}_{r}$ | deg | Wing pitch angles |

$\delta x,\delta y,\delta z$ | mm | Wing hinges translation in the body frame |

${\alpha}_{\ell},{\alpha}_{r}$ | $\mathrm{deg}/\mathrm{mm}$ | Wing twist per mm |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Ben-Dov, O.; Beatus, T.
Model-Based Tracking of Fruit Flies in Free Flight. *Insects* **2022**, *13*, 1018.
https://doi.org/10.3390/insects13111018

**AMA Style**

Ben-Dov O, Beatus T.
Model-Based Tracking of Fruit Flies in Free Flight. *Insects*. 2022; 13(11):1018.
https://doi.org/10.3390/insects13111018

**Chicago/Turabian Style**

Ben-Dov, Omri, and Tsevi Beatus.
2022. "Model-Based Tracking of Fruit Flies in Free Flight" *Insects* 13, no. 11: 1018.
https://doi.org/10.3390/insects13111018