A Gaussian Mixture Model to Separate Birds and Insects in Single-Polarization Weather Radar Data

Abstract

Recent and archived data from weather radar networks are extensively used for the quantification of continent-wide bird migration patterns. While the process of discriminating birds from weather signals is well established, insect contamination is still a problem. We present a simple method combining two Doppler radar products within a Gaussian mixture model to estimate the proportions of birds and insects within a single measurement volume, as well as the density and speed of birds and insects. This method can be applied to any existing archives of vertical bird profiles, such as the European Network for the Radar surveillance of Animal Movement repository, with no need to recalculate the huge amount of original polar volume data, which often are not available.

1. Introduction

Weather radar data are increasingly being used for quantifying the flow of nocturnal bird migration [1,2,3]. In the past, the reflectivity from birds in the air and mean radial velocity has been used to quantify bird biomass [4] before algorithms were developed to extract bird data automatically [5,6,7,8]. These algorithms distinguish between bio-scatterers and precipitation but perform less well in distinguishing between the two main bio-scatterers, birds and insects. To date, these algorithms have mainly been applied to nightly weather radar data during spring and autumn migrations, when birds by far outweigh insects [9,10,11]. Nonetheless, on some nights, large insect masses can also migrate and interfere with, or even mask, bird signals [12,13]. Consequently, classifying and distinguishing insects and birds remains a major challenge [14]. With the progressive replacement of single-polarization weather radar by dual-polarization systems, this classification could be improved considerably [13,15,16]. However, today almost all archived weather radar data consist of single-polarization weather radar data [17,18]. Additionally, the weather radar data publicly available in Europe to ecologists consist only of vertical profiles of bird migration intensities derived from single-polarization weather radar data.

To date, insects are usually filtered using a threshold on the standard deviation of radial velocity and/or an additional threshold on absolute [7,11,19,20,21,22,23,24,25,26]. Here, we combine these features within a Gaussian mixture model to estimate the proportions of birds and insects and their density and speed within a single measurement.

2. Data

2.1. Weather Radar Data

We used the vertical profile time series of reflectivity (cm²/km³), ground speed (m/s), direction (°), and standard deviation of the radial velocity (m/s) generated with the vol2bird software [6,7] from single-polarization polar volumes archived in a publicly accessible repository from the European Network for the Radar surveillance of Animal Movement [18]. We used data from 37 weather radars in France, Germany, the Netherlands, and Belgium operating between 13 February 2018 and 1 January 2019. Each radar provides spatial and temporal data points, with a vertical resolution of 200 m (0–5000 m a.s.l.) and a temporal resolution of 5 min. As we were primarily interested in quantifying nocturnal bird migration, the analysis was limited to night-time, as defined by the local sunrise and sunset at each radar location.

2.2. Data Cleaning

The vertical profiles of reflectivity were manually cleaned using a MATLAB graphical user interface program, which consisted of removing vertical profiles contaminated by precipitation and eliminating altitude bins contaminated by ground scattering (detailed in Appendix A of Nussbaumer et al. [26]). The ground speed values were only kept where reflectivity values were available, thus removing erroneous speeds. While high-reflectivity weather events (i.e., rain) were easily removed during this procedure, slow-moving weather events with low reflectivity (i.e., snow, mist) were difficult to identify visually and therefore were still present at this stage.

2.3. Airspeed

Airspeed was computed by taking the ground speed obtained from weather radar and subtracting the wind speed provided by the ERA5 reanalysis product [27]. The original wind speed data (0.25° × 0.25° × (various pressure level) × 1 h) were linearly interpolated at the location of each data point (radar location, altitude bins of 200 m, 15 min resolution). Pressure levels were converted to altitudes with the standard barometric formula, resulting in a similar resolution of the data points (250 m below 2000 m and 550 m above). The horizontal resolution (0.25° × 0.25° ≈ 18 km × 28 km) of the reanalysis roughly matched the scanning range of the weather radar (25 km). However, the temporal downscaling from 1 h to 15 min can be expected to produce smoother windspeeds. The resulting dataset used in this study consisted of 6.8 million data points.

3. Methodology

The proportion of birds and insects in the sampled volume was estimated by comparing the measured airspeed and standard deviation of the radial velocity to the typical signature produced by birds and insects (Figure 1). This method contrasts with previous approaches that applied fixed thresholds in the standard deviation of the radial velocity (e.g. >2m/s) and/or airspeed (e.g., >5m/s).

First, the typical signature of birds and insects was derived by fitting a two-component Gaussian mixture model to all airspeed and radial velocity standard deviation data points (i.e., one component represents birds and the other insects) (Section 3.1). Realizing that relevant insect-like signals were present during winter, we separated them into either insect- or weather-related signals (e.g., snowfall) based on the timing of their presence (Section 3.2). Then, the fitted model was used to determine the proportion of birds and insects for all data points (Section 3.3). Finally, the bird density (Section 3.4) and the speed of the birds (Section 3.5) were also corrected according to the proportion of birds estimated.

3.1. Model Fitting

The model for estimating the proportion of birds and insects was fitted in two steps.

In the first step, we derived the parameters of the distribution of airspeed and standard deviation of the radial velocity of birds and insects (Figure 1), with a mixture of Gaussian model classification. Assuming that birds and insects fly in a similar manner everywhere, all data points (i.e., all radars, all altitudes, all times) were used to compute the empirical probability density function of airspeed and the standard deviation of radial velocity from the Kernel smoothing function (ksdensity in MATLAB). Then, a two-component Gaussian mixture model was fitted on the empirical probability density function with an expectation-maximization algorithm (fitgmdist on MATLAB). We extracted the location (mean) and shape (covariance) parameters from the two Gaussian components (one for the bird and one for the non-bird).

In the second step, we accounted for spatial and temporal variation in the presence of birds and insects (Figure 2 and Figure 3). The empirical probability density function for each month and each radar was built separately (see Supplementary Material Figure S1). The amplitudes of the two Gaussian components were fitted for each empirical probability density function while keeping the location and shape parameters estimated in the first step. Since the empirical probability distribution function is normalized, the sum of the two Gaussian amplitudes is one. Therefore, a single parameter was fitted (the amplitude ratio), which ranges from 0 (only insects) to 1 (only birds) (Figure 3). Finally, to smoothen the transitions from one month to the next, we temporally interpolated the amplitude ratios of each radar with a shape-preserving piecewise cubic interpolation (pchip on MATLAB) (Figure 4).

3.2. Separation of Insects and Weather

While rain is easily recognized due to its high reflectivity signal and therefore correctly removed in the initial cleaning step, slow-moving weather events with low reflectivity, such as snow and mist (low airspeed and low radial velocity standard deviation) are harder to identify and remove from the dataset (see winter insect-like signal from Germany in Figure 2 and Figure 3). Nevertheless, because snow and mist generally occur in winter [28] and insects mainly occur in summer [29], their corresponding signals can be separated based on their temporal occurrence.

For each radar, the complement of the amplitude ratio (i.e., 1-amplitude ratio) was fitted with the sum of two skewed normal density functions over time, one matching insects (peaking in summer) and the other weather (peaking in winter, Figure 5a). The insect ratio was finally determined as the ratio of the insect normal density function over the sum of the normal density functions (Figure 5b).

3.3. Estimating the Proportion of Birds and Insects

The procedure to estimate the proportion of birds, insects, and remaining weather signal for each data point is as follows. For each data point, the amplitude ratio was computed from the interpolation in Figure 4 based on radar and time of year. Then the two Gaussian distributions were built using both the amplitude ratio and the means and covariances estimated in Figure 1. Next, the probabilities of the bird and non-bird Gaussian probability density functions were determined using the known value of airspeed and radial velocity standard deviation at the data point. Finally, the proportion of birds for that particular data point was estimated by normalizing the probability of birds with the sum of the probabilities of birds and non-birds.

The proportion of insects was determined by multiplying the proportion of non-birds (i.e., 1-proportion of birds) with the insect ratio determined in Figure 5b.

3.4. Bird Density Correction

After quantifying the relative proportion of birds, insects, and weather (in percentage of reflectivity), the bird and insect reflectivity was estimated by multiplying the original values of reflectivity according to their respective proportion. Bird reflectivity was then converted to bird density, assuming a radar cross-section of 11 cm² [6].

3.5. Ground and Air Speed Correction

We propose here a method to correct the ground speed calculated by vol2bird from the contamination of insect movements according to the proportion of birds estimated in Section 3.3.

This problem was formalized with Bayes’ Theorem, where the prior distributions of bird and insect airspeeds are taken from the Gaussian fits of Figure 1 as a normal distribution,

$$B~\mathcal{N}\left({\mu }_B,{\sigma }_B\right)$$

(1)

and

$$I~\mathcal{N}\left({\mu }_I,{\sigma }_I\right)$$

(2)

The airspeed obtained from the weather radar is considered as a mixture of bird airspeed and insect airspeed, weighted according to the proportion of birds $\alpha$ and insects $1-\alpha$, as determined in Section 3.3. Its distribution is therefore also normal, with parameters which can be determined using:

$$\begin{array}{cc}X& =\alpha B+\left(1-\alpha \right)I\\ & ~\mathcal{N}\left({\mu }_X,{\sigma }_X\right)\\ & ~\mathcal{N}\left(\alpha {\mu }_B+\left(1-\alpha \right){\mu }_I,\sqrt{{\alpha }^2{{\sigma }_B}^2+{\left(1-a\right)}^2{\sigma }_I{}^2}\right).\end{array}$$

(3)

In Bayes’ Theorem framework, the corrected airspeed for birds is the probability of bird airspeed conditional to the measured airspeed, shown as:

$$\begin{array}{cc}P\left(B\mid X\right)& =P\left(B\right)\frac{P\left(X\mid B\right)}{P\left(X\right)}\hfill \\ & \propto f_{\mathcal{N}}\left(b\mid {\mu }_B,{\sigma }_B\right)f_{\mathcal{N}}\left(x\mid \alpha b+\left(1-\alpha \right){\mu }_I,{\sigma }_I\right),\hfill \end{array}$$

(4)

where $f_{\mathcal{N}}$ is the normal probability distribution function, $x$ is the known measured airspeed, and $b$ is the unknown bird airspeed.

Each airspeed data point was corrected as follows. For each data point, the probability distribution of the birds’ airspeed $b$, of Equation (4), was built based on the known proportion of birds $\alpha$ and the original airspeed $x$. The corrected airspeed of birds was then sampled randomly according to the probability density function. Finally, the bird ground speed was determined by re-adding the wind speed to the corrected airspeed. The insect ground speed was computed using the sampled bird airspeed and Equation (3).

Figure 6 compares the original airspeed data with the corrected airspeeds for insects and birds. The histograms of the corrected airspeed perfectly reproduce the fitted distribution of insect and bird airspeed by construction. Based on the airspeed distribution, a threshold of 4.8 m/s would lead to the best classification of insects and birds. The traditional 5 m/s threshold would misclassify 19% of bird samples and 4% of insect samples.

4. Results

The results of the Gaussian mixture model fit are:

$${\mathcal{N}}_B=\left(\left[\begin{array}{c}8.0\\ 4.1\end{array}\right],\left[\begin{array}{cc}11.6& -1.2\\ -1.2& 0.9\end{array}\right]\right)$$

(5)

and

$${\mathcal{N}}_I=\left(\left[\begin{array}{c}2.6\\ 2.8\end{array}\right],\left[\begin{array}{cc}1.8& 0.2\\ 0.16& 1.1\end{array}\right]\right)$$

(6)

The resulting corrected dataset is available at https://zenodo.org/record/4587338 [30] accessed on 19 May 2021. Figure 7 illustrates the weekly mean reflectivity and reflectivity traffic rate (RTR) averaged over all radars for birds, insects, and weather-related signals. Weather-related signals account for a small yet significant proportion of the reflectivity in the winter months (December–January).In contrast, insects contribute significantly to the reflectivity from May to mid-September, with, e.g., more than 90% of the reflectivity attributed to insects in June alone.

In comparison, the traditional threshold approach eliminates data points with an airspeed smaller than 5 m/s and a radial velocity standard deviation lower than 2 m/s. The bird reflectivity resulting from this threshold approach (black line in Figure 7) is similar to the proportion approach during the bird migration period (March–May, and mid-Sept–mid-Oct), showing that the known threshold from the literature is well calibrated to the airspeed data. Yet, during summer (mid-May to September), the reflectivity derived from the threshold approach is higher than the one derived from the proportion approach (Figure 1). Data points with an airspeed just below 5 m/s but with high radial velocity standard deviation (~5 m/s) are discarded by the threshold method, while the new approach considers it likely that they are birds.

RTR (reflectivity × speed) (see Supplementary Material Figure S2) assesses the biomass movement (i.e., dynamic instead of static). Due to their lower flight speeds, insects have a smaller proportion in RTR than in reflectivity. However, due to the speed correction, the RTR of birds increases slightly compared to the threshold approach when insects are present (e.g., August–September).

5. Discussion

The method presented in this study exhibits several improvements for separating birds and insects in weather radar signals.

Firstly, in contrast to the previous threshold methods, this approach considers the velocities derived from the radial doppler-speeds as an average resulting from the movements of all objects (including birds and insects) within the scanning range of the radar. Consequently, we suggest estimating the proportion of birds rather than whether insects are present or not.

Secondly, this approach takes advantage of combining both the average airspeed and the variation in airspeed within a single data point to improve the differentiation between birds and insects. Indeed, birds flying with a low mean airspeed (e.g., 3–5 m/s) typically have a high standard deviation in radial velocity (see the skewed orientation of the bird Gaussian contours in Figure 1). This deviation is mainly caused by birds flying in different directions, which results in speeds with low average but high variability. Thus, only if we jointly consider both variables can we achieve better separation.

Finally, a major strength of the method is its capacity to account for the spatio-temporal variation in birds, insects, and low-reflectivity weather events. This is particularly useful when accounting for the presence of snow, which mainly occurs in the north (e.g., Germany) during winter (Figure 3). Finally, the method can also estimate the airspeed of birds by accounting for the contribution of insects to the average airspeed.

Our methodology makes some simplifying assumptions.

Firstly, we calculated the location (mean) and shape (covariance) of the empirical probability density function for the entire dataset, assuming that birds and insects fly with the same distribution of airspeed and standard deviation of radial velocity throughout the year. However, birds typically fly slower and with a more uniform radial velocity (i.e., lower standard deviation) in warmer months (e.g., April, September) than in colder months (i.e., early spring and late autumn) (Figure 2). Indeed, these observations are consistent with the fact that smaller birds (e.g., warblers) with slower optimal airspeeds dominate the long-distance migrations, while larger birds (e.g., thrushes) dominate the short-distance migrations [31]. The variable bird airspeed could be accounted for in future refinements of our methodology with a Gaussian mean function over time.

Secondly, airspeed and the standard deviation of radial velocity are assumed to follow a Gaussian distribution. Despite the imperfect match (Figure 1), Gaussian probability density functions were deemed sufficient for the separation here, although other distributions could be used in the future to improve the distribution fit (with caution, given the increased complexity and likely minimal benefit).

Lastly, we did not consider potential changes in the airspeed distribution (and amplitude) with altitude, as the quality of weather radar at low heights is still questionable [32]. Nevertheless, future improvements could incorporate the information of altitude for separation (see Supplementary Material Figure S3).

6. Conclusions

In this study, we presented a novel method to separate birds and insects in weather radar signals based on the differences between birds and insects in airspeed and standard deviation of radial velocity. This method can be applied to existing single-polarization weather radar data [18] without the need to recalculate a huge amount of original polar volume data. Future work could include the validation of this approach using polarimetric data or a tracking radar.