The Effect of Real-World Interference on CNN Feature Extraction and Machine Learning Classification of Unmanned Aerial Systems

Abstract

Small unmanned aerial systems (UASs) present many potential solutions and enhancements to industry today but equally pose a significant security challenge. We only need to look at the levels of disruption caused by UASs at airports in recent years. The accuracy of UAS detection and classification systems based on radio frequency (RF) signals can be hindered by other interfering signals present in the same frequency band, such as Bluetooth and Wi-Fi devices. In this paper, we evaluate the effect of real-world interference from Bluetooth and Wi-Fi signals concurrently on convolutional neural network (CNN) feature extraction and machine learning classification of UASs. We assess multiple UASs that operate using different transmission systems: Wi-Fi, Lightbridge 2.0, OcuSync 1.0, OcuSync 2.0 and the recently released OcuSync 3.0. We consider 7 popular UASs, evaluating 2 class UAS detection, 8 class UAS type classification and 21 class UAS flight mode classification. Our results show that the process of CNN feature extraction using transfer learning and machine learning classification is fairly robust in the presence of real-world interference. We also show that UASs that are operating using the same transmission system can be distinguished. In the presence of interference from both Bluetooth and Wi-Fi signals, our results show 100% accuracy for UAV detection (2 classes), 98.1% (+/−0.4%) for UAV type classification (8 classes) and 95.4% (+/−0.3%) for UAV flight mode classification (21 classes).

1. Introduction

Deloitte Economics recently reported that UASs could tackle issues in industry problems, such as logistics and road congestion [1]. In the UK, the Royal Mail have started a trial using UASs for mail delivery to remote parts of the island [2]. The current scope of the benefits UASs could bring to the economy has been branded limitless. However, in February 2021, concerns were raised over small UASs that could be purchased at retail shops such as Costco and the need for dependable countermeasures [3]. Nassi et al. [4] consider societal threats, in particular those surrounding security and privacy for areas that allow UAS flights. Nassi et al. highlighted the ‘detection purpose problem’ asking the question, how do you know whether a UAS flying over a back garden is delivering a pizza or being used for illegitimate purposes? They suggest ways of authenticating UASs and UAS operators. Altawy and Youssef [5] consider security, safety and privacy issues raised by UAS use in national airspace and conclude that security will enable the next generation of UAS. Lykou et al. [6] consider asymmetric threats to Airports from UASs in three categories: spying; carrying chemical radiological, biological, nuclear and explosive materials; and conducting cyber-attacks. Since 2016, a number of security incidents have occurred around airports and other buildings deemed critical national infrastructure.

The Department for Transport, the Military Aviation Authority and the British Airline Pilots’ Association commissioned a study to consider whether critical damage (major structural damage) could occur from mid-air collisions between UASs and manned aircraft. The report showed that helicopter windscreens, especially those which were not bird strike certified, were vulnerable to UASs weighing 400 g, as were tail rotors. Passenger airlines were more resilient, showing critical damage to occur with 2 kg UASs, but in both cases, the structure of the UAS also had an effect on the damage [7]. The last UK AIRPROX board summary for 2019 reported 125 aircraft to small UAS incidents with a 62% chance of an encounter with a small UAS being risk bearing [8]. Incidents with small UASs are not always illegitimate; for example, a small UAS being used for demonstration purposes at Goodwood Aerodrome lost control and crashed close to an active airport airspace [9]. The largest incident to disrupt airport activities happened in 2018 at Gatwick Airport.

Table 1. UAS Airport Disruptions.

Date	Location	Observation	Disruption
2021	Auckland Airport, New Zealand [10]	Pilot sighting of drone 30 m from helicopter and 5 m above	Flights grounded 15 min
2021	Piedmont Triad International Airport, North Carolina Airport, USA [11]	Drone sightings over airport	Flights suspended 2 h, 1 flight diverted
2020	Adolfo Suárez-Barajas Airport, Spain [12]	2 pilot sightings of drones	Flights grounded 1 h, 26 flights diverted
2020	Frankfurt Airport, Germany [13]	Pilot sighting of drone	Flights grounded 2 h, flights diverted and cancelled
2020	Stansted Airport, UK [14]	Military helicopter confirmed sighting of drone	No flight disruption, one police arrest made
2019	Changi Airport, Singapore [15]	Drone sightings in vicinity of airport	37 flights delayed, 1 flight diverted
2019	Dubai Airport [16]	Drone sightings in vicinity of airport	30 min suspension of flights
2019	Dublin Airport, Ireland [17]	Pilot sighting of drone	Flights grounded 30 min, 3 flights diverted
2019	Frankfurt Airport, Germany [18]	Sighting of drone	Flights grounded 1 h, 100 take-offs and landings were cancelled
2019	Frankfurt Airport, Germany [19]	Sighting of drone	Flights grounded 30 min
2019	Heathrow Airport, UK [20]	Undisclosed number of sightings	Flights grounded for 1 h
2019	Heathrow Airport, UK [21]	Heathrow Pause group planned drone flights to disrupt flights	1 attempted flight which was unsuccessful
2019	Kansai International Airport, Japan [22]	Drone sighted hovering near terminal and flying over runway 1 week prior aircrew sighting of drone in vicinity of incoming aircraft	1 h suspension flights 40 min suspension of flights
2019	Newark Airport, USA [23]	2 pilot sightings on route into Newark, above Teterboro airport, drone coming within 9 m of aircraft	Flights disrupted for short duration
2018	Gatwick Airport, UK [24]	170 sightings, 115 sightings deemed credible	Airport closed for 33 h, 1000 flights cancelled, 140,000 passengers affects at a cost of GBP 50 million. 18 month police operation costing GBP 800,000 across 5 different forces.

shows a selection of events that have disrupted airports since December 2018, but this is by no means an exhaustive list.

A number of different ways of detecting and classifying UASs are being researched and implemented today, including a number of significant European Projects. The European ALADDIN project integrates RADAR, acoustic and optical sensors to perform detection, classification and localisation with a command and control node. The project boasts an integrated system that counters malicious UASs [25]. Another project called SafeShore aims to detect UAS launching from boats and people in the sea. The project seeks to achieve this using three-dimensional LIDAR integrated with acoustic, radio and imagery data [26]. While current research and development programs, such as SafeShore and ALADDIN, look to combine and integrate various types of detection and classification methods, each individual method is still advancing with current research.

With respect to current literature we will first consider audio detection and classification methods. Many commercial UASs have an audio signature at 200 Hz, and detection methods have been evaluated using microphone arrays. Mezei et al. [27] use correlation with varied results, improving with the use of high-quality microphones and a larger sampling rate. We note that this technique succeeded with a technique using feature extraction and machine learning techniques in [28,29]. This method is short range at 100 m [27] and has shown to be susceptible to noise [30]. Shi et al. [29] design a detection and localisation system using audio signals and an array of microphones. The system works in real time and uses TDOA and a Bayesian filter for localisation.

Another well-documented detection and classification method is the use of video and imagery data. Thai et al. [31] use a CNN for feature extraction of motion patterns and k-nearest neighbour (kNN) to classify results. Schumann et al. [32] use a convolutional neural network (CNN) to detect UASs for the birds vs. drones competition. They find that when they increase the training data to include images from the web from many varied scenarios, their results increase in accuracy. The birds vs. drones challenge [33] is set up because drones can be confused with birds. As such, false detection rates have shown to be high with video detection methods [30]. Coluccia et al. [34] consider the results of the 2020 drones vs. birds competition and in particular consider the top three performing algorithms. They observed the biggest challenges to be detection at a distance and moving cameras, recommending that training and test data be expanded to include this. Further, discrimination between birds and UASs becomes considerably harder with distance.

RADAR has been shown to be very effective at detecting aircraft across large distances but has failed when trying to detect smaller UASs. This is determined to be due to their size and lack of noise and signal emissions [30]. Mendis et al. [35] use a radar sensor and a deep belief network (DBN) to classify three types of micro UASs with an accuracy of over 90%, but the paper only considered a lab environment. Zulkifli and Balleri [36] design and develop a radar system to detect nano-drones using a micro-Doppler technique. Semkin et al. [37] consider radar cross-sections for detection and classification in urban environments and suggest further work to include the use of machine learning. Practical issues with RADAR detection systems include the cost of implementation and the requirement for licenses for transmission [6]. Andraši et al. [38] consider thermal detection using heat signatures and found that engines that were electrically powered (compared to fuel powered) only gave off a very small signature as they were so efficient. However, the heat from the batteries instead of the motors could provide a signature for detection at night time, but a human in the loop was required due to noise interference. Yaacoub et al. [30] suggest that thermal detection systems are associated not only with high cost, like with RADAR, but short detection distances and low detection accuracy. Coluccia et al. [39] provide a comprehensive review of radar systems for UAS detection and classification. They consider the main challenges relating to detection, verification and classification. They also focus on the latest technology with respect to frequency modulated continuous wave (FMCW) radar and spatially distributed sensors. Passafiume et al. [40] investigate the reliability of FMCW radar images for the classification of a UAS by the amount of motors it has, and they rule out the rotation speed from effecting classification.

RF detection systems boast advantages such as range, cost and the ability to perform triangulation to determine location. RF systems in conjunction with machine learning have proven successful in recent research [41,42,43,44,45]. RF has its own challenges as with the other detection methods previously discussed. It requires an active RF signal with the controller, and it suffers from interference in congested RF environments [6]. This paper extends previous work conducted by Swinney and Woods [44,45] by expanding the open ‘DroneDetect’ dataset [46] to 21 classes for 7 UAS types and flight modes to include hovering, switched on but not taken off and flying. Secondly, the work on interference is extended to the evaluation of environments where both Bluetooth and Wi-Fi signals are present concurrently. The paper is constructed as follows: Section 2 discusses the methodology, Section 3 the results and Section 4 the conclusions.

2. Materials and Methods

2.1. Experimental Setup

The experimental setup can be seen in Figure 1, and it was used to create the DroneDetect dataset [46]. The yellow box shows the laptop that is running GNURadio. GNURadio is an open-source toolkit designed for signal processing with SDRs [47], and it was used to capture the signals.

The SDR used in our experiments was the BladeRF by Nuand. It has a frequency range of 47 MHz to 6 GHz and is low cost at USD 480 [48]. The Palm Tree Vivaldi Antenna [49,50] with a frequency range of 800 MHz to 6 GHz was chosen due to its small, portable and low cost ($18.99 [51]) attributes. Signals were recorded with a centre frequency of 2.4375 GHz and a sample rate of 60 MBits/s over a 28 MHz bandwidth. Each sample was the result of 20 ms of recording time in complex form, and the dataset comprised of 500 samples per class.

Table 2. UAS Classes.

Classification Type	Class	Description
Detection	1	No UAS detected
Detection	2	UAS detected
midrule Type	1	No UAS detected
Type	2	Mavic 2 Air S detected
Type	3	Parrot Disco detected
Type	4	Inspire 2 Pro detected
Type	5	Mavic Pro detected
Type	6	Mavic Pro 2 detected
Type	7	Mavic Mini detected
Type	8	Phantom 4 detected
midruleFlight Mode	1	No UAS detected
Flight Mode	2	Air Mode 1—Switched on
Flight Mode	3	Air Mode 2—Hovering
Flight Mode	4	Air Mode 3—Flying
Flight Mode	5	Disco Mode 1—Switched on
Flight Mode	6	Disco Mode 3—Flying
Flight Mode	7	Inspire Mode 1—Switched on
Flight Mode	8	Inspire Mode 2—Hovering
Flight Mode	9	Inspire Mode 3—Flying
Flight Mode	10	Mavic 1 Mode 1—Switched on
Flight Mode	11	Mavic 1 Mode 2—Hovering
Flight Mode	12	Mavic 1 Mode 3—Flying
Flight Mode	13	Mavic Pro 2 Mode 1—Switched on
Flight Mode	14	Mavic Pro 2 Mode 2—Hovering
Flight Mode	15	Mavic Pro 2 Mode 3—Flying
Flight Mode	16	Mini Mode 1—Switched on
Flight Mode	17	Mini Mode 2—Hovering
Flight Mode	18	Mini Mode 3—Flying
Flight Mode	19	Phantom 4 Mode 1—Switched on
Flight Mode	20	Phantom 4 Mode 2—Hovering
Flight Mode	21	Phantom Mode 3—Flying

describes the classes evaluated over a mix of quadcopter and fixed-wing UAS. To capture the class ‘switched on’, the UAS and controller were placed 4 m from the antenna on the ground. Hovering was conducted at an altitude of 20 m and flying at an altitude of 20 m within a radius of 40 m around the antenna. Bluetooth interference was added by playing music through an end device (JBL Charge Bluetooth Speaker) placed 2 m from the antenna and the connecting phone next to the antenna. Wi-Fi interference was added by playing a YouTube video over a phone hotspot, end device (Apple MacBook) placed 2 m from the antenna and a connected phone hotspot next to the antenna. Both interference sources were active during dataset collection.

2.2. Graphical Signal Representation

For each 20 ms sample, a power spectral density (PSD) and spectrogram were plotted using Python3 MatplotLib using 1024 FFT, a Hanning window and a 120 overlap. PSD is a representation in the frequency domain considering the power distribution at various frequencies. The spectrogram provides a representation of the power of the signal in the time domain. Considering the signal in different graphical representations allows us to benefit from transfer learning, a technique shown by Swinney and Woods in [44,45]. Figure 2 shows a capture of the spectrum in the time domain where no UAS signals are present, only the interference. The yellow bursts of horizontal activity represent the Bluetooth signal, and the larger vertical band spanning the lower end of the frequency range at the beginning of the time frame is the Wi-Fi signal.

Figure 3 shows the same data represented in the frequency domain through a PSD.

Figure 4 and Figure 5 show Phantom 4 when it is switched on in the presence of Bluetooth and Wi-Fi interference in spectrogram and PSD representations, respectively. We can see some clear activity in the lower end of the spectrum.

If we compare Figure 5 with Figure 3, we can see there is a distinct difference between the shape of the signals. Figure 6 and Figure 7 show Phantom 4 hovering at an altitude of 20 m. If we compare Figure 6 with Figure 4, we can observe an increase in activity in yellow.

Comparing Figure 7 to Figure 5, we can see two spikes in activity compared to three in Figure 7 at the centre frequency. Figure 8 and Figure 9 show Phantom 4 flying. It is clear from looking at Figure 9 compared to Figure 5 and Figure 7 that there is another increase in the number of spikes in power at different frequencies.

Now, we move on from Phantom 4, which is a quadcopter-style UAS, to the Parrot Disco, which is fixed wing. For the Disco, we only have two flight modes, as it is not capable of hovering. Figure 10 and Figure 11 shows the Disco switched on in spectrogram and PSD representations, respectively.

When we compare the Disco switched on in Figure 10 to the Phantom 4 in Figure 4, we can see that the disco is a more concentrated signal with less bursts of activity. However, when we compare Figure 11 to Figure 5, we can see a visible difference in the frequency domain.

Figure 12 and Figure 13 shows the Disco flying. We see an increase in power across the whole band in Figure 11 compared to that in Figure 10 and also with the PSD in Figure 13 compared to Figure 11. It is important to note that what we are observing in the RF spectrum is the transmission system.

Table 3. UAS Transmission Systems.

UAS Type	Transmission System
Mavic 2 Air S	OcuSync 3.0
Parrot Disco	Wi-Fi
Inspire 2 Pro	Lightbridge 2.0
Mavic Pro	OcuSync 1.0
Mavic Pro 2	OcuSync 2.0
Mavic Mini	Wi-Fi
Phantom 4	Lightbridge 2.0

lists the UASs we are looking at against their respective transmission systems.

The DJI range of UASs use three types of transmission system: Wi-Fi, Lightbridge and OcuSync. The Mavic Mini uses Wi-Fi and operates in the 5.8 GHz range (5.725–5.850 GHz) and in the 2.4 GHz range (2.4–2.4835 GHz), with an effective isotropic radiated power (EIRP) or transmission power of 19 dBm at 2.4 GHz [52]. The Lightbridge 2 has a maximum (interference free) transmission distance of 5 km, the EIRP of the antenna is 100 mW at 2.4 GHz, and it can operate in the 5.8 GHz range (5.725–5.825 GHz) and the 2.4 GHz range (2.4–2.483 GHz) [53]. The OcuSync scans the band for any interference and decides which transmission channel is best. It then automatically switches between channels during the flight. OcuSync 1.0 and 2.0 have a range of 7 km; they differ due to Ocusync 2.0 being able to utilise 2.4 and 5.8 GHz frequency diversity. OcuSync 1.0 has an EIRP of 26 dBm [54] and OcuSync 2.0 an EIRP of 25.5 dBm. The OcuSync 3.0 range is increased again to 12 km [55] and EIRP 26 dBm [56]. The Parrot Disco uses the SkyController 2, which is a Wi-Fi-based protocol with a range of 2 km using MIMO antennas in the 2.4 and 5.0 Ghz frequency bands [57].

2.3. Classification

Once the datasets of spectrogram and PSD graph images were created and saved as 224 × 224 pixels, a pre-trained VGG16 CNN on ImageNet [58] was used as a feature extractor. Through a process of transfer learning, neural networks trained for one purpose can be used for another purpose. ImageNet is an object detection database consisting of millions of images. Using a pre-trained VGG-16, we are able to utilise the process of transfer learning. This means we do not need to train our weights from scratch, a procedure which needs a massive amount of data and time.

Table 4. VGG-16 CNN.

Layer Type	Size	Feature Map
Input Image	224 × 224 × 3	1
2× Convolutional	224 × 224 × 64	64
Max Pooling	112 × 112 × 64	64
2× Convolutional	112 × 112 × 128	128
Max Pooling	56 × 56 × 128	128
2× Convolutional	56 × 56 × 256	256
Max Pooling	28 × 28 × 256	256
3× Convolutional	28 × 28 × 512	512
Max Pooling	14 × 14 × 512	512 v
3× Convolutional	14 × 14 × 512	512
Max Pooling	7 × 7 × 512	512 v

shows the structure of a VGG-16 CNN [59].

At the end of the last pooling layer, we are left with 25,088 features. These features are then used to train machine learning classifiers, logistic regression (LR) and k-nearest neighbour (kNN). LR works by fitting an ‘S’ shaped curved, otherwise known as a Sigmoid function, to our features. The equation representing the sigmoid function can be seen in Equation (1).

$$\begin{array}{cc}\hfill & sigmoid\left(x\right)=1/(1+e^{-x})\hfill \end{array}$$

(1)

LR estimates the discrete value based on a set of independent variables. It estimates the probability of occurrence by measuring the relationships between one (or more) of those variables and a categorical dependant variable. Limited-memory Broyden–Fletcher–Goldfarb– Shanno was chosen as the solver for our experiments as it deals with multiclass due to its ability to handle multinomial loss [60]. kNN, on the other hand, uses the dataset to find the closet point to the input point. The classifier then works by using a majority vote of neighbours. The Minkowski distance was utilisied as part of our experiments and can be calculated as show in Equation (2).

$$\begin{array}{cc}\hfill & {sum\left(\right|x-y|}^p{)}^{(1/p)}\hfill \end{array}$$

(2)

The k-nearest neighbour of the particular data point is then found and assigned to the class that has the highest probability.

$$\begin{array}{cc}\hfill & P(y=j\mid X=x)=\frac{1}{K}\sum _{iϵA}I(y^{\left(i\right)}=j)\hfill \end{array}$$

(3)

Equation (3) shows the probability of an input x being assigned to the class that has the highest probability [61]. For each of the models, fivefold cross validation was used to indicate whether the model was overfitting. Hyper-parameter regularisation for LR and the number of neighbours for kNN were optimised using threefold nested cross-validation. The same process was followed for classification as described in previous work, and more information can be found in the references [44,45].

Accuracy and F1-score were used to evaluate the models’ performances. To understand both metrics, we first must understand that a true positive is when the algorithm predicted, for example, that Phantom 4 was present and it was there in the data. A true negative is where the prediction was no Phantom 4 and the prediction was correct or true. A false positive is where the algorithm thinks Phantom 4 was there but it was not, and, lastly, the false negative predicts that it was not there but it actually was. Accuracy and F1-score can now be defined. Accuracy is show in Equation (4).

$$\begin{array}{cc}\hfill & Accuracy=\frac{TruePositive+TrueNegative}{TruePositive+TrueNegative+FalsePositive+FalseNegative}\hfill \end{array}$$

(4)

F1-score includes the metrics precision and recall. Equation (5) shows the formula for precision showing how many positive predictions were correct.

$$\begin{array}{cc}\hfill & Precision=\frac{TruePositive}{TruePositive+FalsePositive}\hfill \end{array}$$

(5)

Equation (6) shows the equation for recall and considers the correctly predicted positives as a fraction.

$$\begin{array}{cc}\hfill & Recall=\frac{TruePositive}{TruePositive+FalseNegative}\hfill \end{array}$$

(6)

F1-score can then be calculated as seen in Equation (7).

$$\begin{array}{cc}\hfill & \mathrm{F}1-\mathrm{score}=\frac{2(Precision\ast Recall)}{Precision+Recall}\hfill \end{array}$$

(7)

3. Results

Table 5. Results accuracy (%) and F1-score (%).

Classifier		Metric	Detection	Type	Flight
LR	PSD	Acc	100(+/−0.0)	98.1 (+/−0.4)	95.4 (+/−0.3)
	PSD	F1	100(+/−0.0)	98.1 (+/−0.4)	95.4 (+/−0.3)
	Spec	Acc	96.7 (+/−1.5)	90.5 (+/−0.8)	87.3 (+/−0.4)
	Spec	F1	96.7 (+/−1.5)	90.5 (+/−0.9)	87.3 (+/−0.4)
kNN	PSD
	PSD	Acc	99.6 (+/−0.2)	93.5 (+/−0.6)	86.5 (+/−0.5)
	PSD	F1	99.6 (+/−0.2)	93.4 (+/−0.7)	86.3 (+/−0.5)
	Spec	Acc	88.0 (+/−1.3)	75.1 (+/−1.5)	64.6 (+/−0.9)
	Spec	F1	87.9 (+/−1.4)	75.3 (+/−1.5)	64.8 (+/−0.8)

shows that in the presence of real-world interference Bluetooth and Wi-Fi signals, we can still detect a UAS with 100% accuracy and 100% F1-score using LR and PSD graphical representation. UAS-type classification produces 98.1 (+/−0.4)% accuracy and a 98.1 (+/−0.4)% F1-score, with PSD and LR and UAS flight mode classification achieving 95.4 (+/−0.3)% accuracy and a 95.4 (+/−0.3)% F1-score.

Accuracy and F1-score decrease as the classes increase, but high accuracy is maintained for flight mode classification. The table shows that LR outperforms kNN in all the experiments. Time domain features from the spectrogram graphical signal representations are less robust to the interference than frequency domain PSD features are.

Figure 14 shows the confusion matrix for UAS-type classification. We can observe that the classifier has some misclassification between the Mavic Pro and the Mavic 2 Pro. If we go back to Table 3, we can see that the Mavic Pro uses OcuSync 1.0 and that the Mavic 2 Pro uses OcuSync 2.0. The main difference between the two transmission systems is that OcuSync 2.0 utilises both the 2.4 and 5.8 GHz frequency bands. The misclassification is likely due to the similar nature of the systems in the 2.4 GHz band.

Figure 15 shows the confusion matrix for 21 class UAV flight mode classification. We can observe that the misclassification occurs again between the classes of Mavic Pro and Mavic Pro 2, which we can put down to the similarities between OcuSync 1.0 and 2.0. The second area of misclassification occurs between Phantom 4 switched on and Phantom 4 flying. However, the misclassification is small.

4. Discussion

Overall, we have shown that although UASs can prove a serious security challenge, especially in airfield scenarios, detection and classification can be achieved amongst real-world interference. Using CNN feature extraction with transfer learning and machine learning classifiers, UASs operating with the same transmission systems can be distinguished amongst concurrent Bluetooth and Wi-Fi signals. For UAS detection, 100% accuracy can be achieved, and for UAS types and flight mode classification, values of 98.1% (+/−0.4%) and 95.4% (+/−0.3%), respectively, are achieved. Future work should consider more than one UAS of the same type entering the airspace to evaluate how specific the neural network feature extraction is. Further to this, metrics such as detection distance and detection time can be applied for a trained model to detect and classify a UAS in real time to understand real-world feasibility.