RL-Based Detection, Tracking, and Classification of Malicious UAV Swarms through Airborne Cognitive Multibeam Multifunction Phased Array Radar
Abstract
:1. Introduction
- Common UAVs cannot carry a large multifunction radar due to weight and power constraints. To overcome this limitation, we propose an approach that employs multiple small MMPARs carried by UAVs. These radar nodes are networked to provide a cumulative radar response, allowing us to detect, track, and classify malicious UAVs in a swarm.
- By using multiple MMPAR nodes, our approach eliminates the risk of a single-point failure due to malfunction or external jamming.
- Using multiple MMPAR nodes onboard UAVs provides superior spatial coverage and mobility compared to a single radar node on the ground.
- The use of multiple radar beams in MMPAR and the ability to adaptively schedule the beams help to accurately resolve multiple targets in the range, Doppler, and angular domains. This is because the multiple beams provide better spatial coverage and resolution, while adaptive scheduling ensures that the beams are directed towards areas of interest, where potential targets may be present. Additionally, the use of multiple beams also helps to mitigate the effects of clutter and interference, which can degrade radar performance.
- The multifunction beams generated by the MMPAR can be used for other tasks in addition to the main radar task. For example, the beams can be used for communication purposes or for RF passive listening, which can be performed simultaneously by sharing the radar resources adaptively. This allows for the efficient utilization of the MMPAR resources, enabling the UAVs to perform multiple tasks with a single device.
- Anomaly detection and removal is an important aspect of our approach, as it helps to reduce false alarms and improve the accuracy of the system. By using RL, the system is able to learn from its environment and adjust its parameters to improve its performance. This can help to identify and remove anomalies in real time, leading to the more reliable and efficient detection, tracking, and classification of UAVs.
- Our approach works optimally in complex and dynamically changing scenarios, e.g., UAV swarms, clutter, and jamming.
2. Multibeam Multifunction Phased Array Radar Onboard UAVs
2.1. Steering and Scheduling of Multifunction Phased Array Beams
- There are two communication beams, as shown in Figure 1 and Figure 2. One communication beam facilitates communication between the UAVs and is always available with the highest scheduling priority. The other communication beam forms a link between the ground station (GS) and the MMPAR onboard the UAV, allowing the MMPAR to function as a communication relay. Both communication beams and the volume search beam operate simultaneously during a given . However, when a target is detected by the volume search beam, the second communication beam ceases operation, and communications from the second beam are transferred to other GS nodes.
- To improve the detection and tracking of UAVs, the volume search beam is designed with the largest coverage among all the beams. This enables the radar to scan a vast volume of space during the given time interval . However, once a target is detected, the scheduling priority of the volume search beam corresponding to that target is reduced for future update intervals. This approach ensures that the other beams with higher scheduling priority can focus on tracking the target with more accuracy and efficiency.
- The cued search beam is used to confirm the presence of a target detected by the volume search. Both the cued search and volume search beams can scan simultaneously.
- The tracking beam is directed towards the coordinates provided by the cued search beam after a time interval of . As the coverage area of the track beam is significantly larger than the physical tracks covered by a target during , the target remains within the coverage area. Due to its high angular and range resolution, the tracking beam can distinguish between multiple targets in the range and angular bins, and estimate their position, RCS, and Doppler estimates. Additionally, the tracking beam has a higher priority (after confirming target presence) than other beams. Targets are tracked adaptively, meaning that mobile/fast-moving targets are visited frequently compared to static/slow targets to update their state. Figure 3 illustrates the interaction of different types of beams with targets.
- The passive RF listener beam is not scheduled in a fixed sequence but rather is randomly applied at different time intervals of . Its purpose is to detect active RF emissions from a malicious UAV, including intentional RF emissions for jamming and the RF link of the UAV for further analysis. Figure 3 illustrates the passive RF listener beam that is used to detect directed RF emissions from a malicious UAV.
Algorithm 1 Pseudo-code for beam scheduler and working of MMPAR |
|
2.2. MMPAR Parameters
3. Reinforcement Learning in Our Approach
3.1. State of the Targets
3.2. Anomalies
Algorithm 2 Pseudo-code for multi-agent RL algorithm using Q-learning and MMPAR data |
|
Anomalies | States | Reward for | |||
---|---|---|---|---|---|
Anomaly not removed | [ | ⋯ | |||
Anomaly (1) | Anomaly removed but reappears after | [ | ⋯ | ||
Anomaly removed and does not reappear | [ | ⋯ | |||
⋮ | ⋮ | ⋮ | ⋮ | ⋯ | ⋮ |
Anomaly not removed | [ | ⋯ | |||
Anomaly (6) | Anomaly removed but reappears after | [ | ⋯ | ||
Anomaly removed and does not reappear | [ | ⋯ |
Anomaly (Observed from Target Returns after Duration ) | Actions | Q-Values | |
---|---|---|---|
↓ | |||
↑ | |||
Low number of targets detected | |||
↓ | |||
↑ | |||
Detection SNR below threshold | ↑ | ||
Velocity changes above threshold | ↑ | ||
Scenario | |||
↓ | |||
↑ | |||
RCS changes above threshold | ↓ | ||
Active RF emissions from a UAV | ↓ | ||
↓ | |||
↑ | |||
Classification of targets changing | ↑ | ||
↑ | |||
↓ | |||
Algorithm 3 Pseudo-code for evaluation and comparison of RL-based approach with other AI algorithms |
|
3.3. Multi-Agent Q-Learning
3.4. Anomaly Removal Using Supervised AI Algorithms
4. Simulation Setup and Results
4.1. Simulation Setup
4.2. Results and Analysis
4.3. Comparison of RL with Other AI Algorithms
5. Conclusions and Future Work
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Khawaja, W.; Guvenc, I.; Matolak, D.W.; Fiebig, U.C.; Schneckenburger, N. A Survey of Air-to-Ground Propagation Channel Modeling for Unmanned Aerial Vehicles. IEEE Commun. Surv. Tutor. 2019, 21, 2361–2391. [Google Scholar] [CrossRef] [Green Version]
- Hassanalian, M.; Abdelkefi, A. Classifications, applications, and design challenges of drones: A review. Progress Aerosp. Sci. 2017, 91, 99–131. [Google Scholar]
- Beloev, I.H. A review on current and emerging application possibilities for unmanned aerial vehicles. Acta Technol. Agric. 2016, 19, 70–76. [Google Scholar] [CrossRef] [Green Version]
- Alghamdi, Y.; Munir, A.; La, H.M. Architecture, Classification, and Applications of Contemporary Unmanned Aerial Vehicles. IEEE Consum. Electron. Mag. 2021, 10, 9–20. [Google Scholar] [CrossRef]
- Custers, B. Future of Drone Use; Springer: Berlin/Heidelberg, Germany, 2016. [Google Scholar]
- Khawaja, W.; Semkin, V.; Ratyal, N.I.; Yaqoob, Q.; Gul, J.; Guvenc, I. Threats from and countermeasures for unmanned aerial and underwater vehicles. Sensors 2022, 22, 3896. [Google Scholar] [CrossRef] [PubMed]
- Dani, R.R.; Kubde, R.R.; Sadhu, M.A. A Comprehensive Study of Various Techniques for Hostile Drone Detection and Their Classification. In Proceedings of the IEEE International Conference on Communication and Electronics Systems, Coimbatore, India, 8–10 July 2021; pp. 1538–1543. [Google Scholar]
- Shi, X.; Yang, C.; Xie, W.; Liang, C.; Shi, Z.; Chen, J. Anti-Drone System with Multiple Surveillance Technologies: Architecture, Implementation, and Challenges. IEEE Commun. Mag. 2018, 56, 68–74. [Google Scholar] [CrossRef]
- Ezuma, M.; Ozdemir, O.; Anjinappa, C.K.; Gulzar, W.A.; Guvenc, I. Micro-UAV Detection with a Low-Grazing Angle Millimeter Wave Radar. In Proceedings of the IEEE Radio and Wireless Symposium (RWS), Orlando, FL, USA, 20–23 January 2019; pp. 1–4. [Google Scholar]
- Hoffmann, F.; Ritchie, M.; Fioranelli, F.; Charlish, A.; Griffiths, H. Micro-Doppler based detection and tracking of UAVs with multistatic radar. In Proceedings of the IEEE Radar Conference, Philadelphia, PA, USA, 2–6 May 2016; pp. 1–6. [Google Scholar]
- Jian, M.; Lu, Z.; Chen, V.C. Drone detection and tracking based on phase-interferometric Doppler radar. In Proceedings of the IEEE Radar Conference, Oklahoma City, OK, USA, 23–27 April 2018; pp. 1146–1149. [Google Scholar]
- Aldowesh, A.; Shoaib, M.; Jamil, K.; Alhumaidi, S.; Alam, M. A passive bistatic radar experiment for very low radar cross-section target detection. In Proceedings of the IEEE Radar Conference, Johannesburg, South Africa, 27–30 October 2015; pp. 406–410. [Google Scholar]
- Yao, Y.; Zhao, J.; Wu, L. Waveform optimization for target estimation by cognitive radar with multiple antennas. Sensors 2018, 18, 1743. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Haykin, S. Cognitive radar: A way of the future. IEEE Signal Process. Mag. 2006, 23, 30–40. [Google Scholar] [CrossRef]
- Metcalf, J.; Blunt, S.D.; Himed, B. A machine learning approach to cognitive radar detection. In Proceedings of the IEEE Radar Conference, Johannesburg, South Africa, 27–30 October 2015; pp. 1405–1411. [Google Scholar]
- Smith, G.E.; Gurbuz, S.Z.; Brüggenwirth, S.; John-Baptiste, P. Neural Networks & Machine Learning in Cognitive Radar. In Proceedings of the IEEE Radar Conference, Virtual, 21–25 September 2020; pp. 1–6. [Google Scholar]
- Thornton, C.E.; Kozy, M.A.; Buehrer, R.M.; Martone, A.F.; Sherbondy, K.D. Deep Reinforcement Learning Control for Radar Detection and Tracking in Congested Spectral Environments. IEEE Trans. Cognitive Commun. Netw. 2020, 6, 1335–1349. [Google Scholar] [CrossRef]
- Ahmed, A.M.; Ahmad, A.A.; Fortunati, S.; Sezgin, A.; Greco, M.S.; Gini, F. A Reinforcement Learning Based Approach for Multitarget Detection in Massive MIMO Radar. IEEE Trans. Aerosp. Electron. Syst. 2021, 57, 2622–2636. [Google Scholar] [CrossRef]
- Yang, T.; De Maio, A.; Zheng, J.; Su, T.; Carotenuto, V.; Aubry, A. An Adaptive Radar Signal Processor for UAVs Detection With Super-Resolution Capabilities. IEEE Sens. J. 2021, 21, 20778–20787. [Google Scholar] [CrossRef]
- Wu, Q.; Wang, H.; Li, X.; Zhang, B.; Peng, J. Reinforcement learning-based anti-jamming in networked UAV radar systems. Appl. Sci. 2019, 9, 5173. [Google Scholar] [CrossRef] [Green Version]
- Chen, Y.J.; Chang, D.K.; Zhang, C. Autonomous Tracking Using a Swarm of UAVs: A Constrained Multi-Agent Reinforcement Learning Approach. IEEE Trans. Veh. Technol. 2020, 69, 13702–13717. [Google Scholar] [CrossRef]
- Mendis, G.J.; Wei, J.; Madanayake, A. Deep learning cognitive radar for micro UAS detection and classification. In Proceedings of the IEEE Cognitive Communications for Aerospace Applications Workshop (CCAA), Cleveland, OH, USA, 27–28 June 2017; pp. 1–5. [Google Scholar]
- Wang, W.; Zhou, X.; Zhang, B.; Mu, J. Anomaly detection in big data from UWB radars. Secur. Commun. Netw. 2015, 8, 2469–2475. [Google Scholar] [CrossRef]
- Li, J.; Dani, H.; Hu, X.; Liu, H. Radar: Residual Analysis for Anomaly Detection in Attributed Networks. In Proceedings of the IJCAI, Melbourne, Australia, 19–25 August 2017; Volume 17, pp. 2152–2158. [Google Scholar]
- Yu, T.; Sun, Y.; Nanda, S.; Sekar, V.; Seshan, S. RADAR: A Robust Behavioral Anomaly Detection for IoT Devices in Enterprise Networks (CMU-CyLab-19-003). 2019. Available online: https://kilthub.cmu.edu/articles/report/RADAR_A_Robust_Behavioral_Anomaly_Detection_for_IoT_Devices_in_Enterprise_Networks_CMU-CyLab-19-003_/16709533 (accessed on 20 May 2023).
Functionalities | Our Approach | [18] | [19] | [20] | [21] | [22] |
---|---|---|---|---|---|---|
Phased array multifunction, multibeam steering (simultaneously) and adaptive beam scheduling | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
Airborne and mobility in air | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ |
Networked radar nodes | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ |
Multi-agent RL | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ |
Anomalies in detection, tracking, and classification identification and removal | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
Cognitive or adaptive adjustment of radar parameters or radar platform position adjustment | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Multi-target detection/tracking/ classification | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
Performance degradation in complex and dynamic environments | No | No | Not reported | Not reported | No | Not reported |
Communications relaying capability in addition to radar operation | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
Serial # | Radar Parameter | Representation |
---|---|---|
1 | Transmit power | |
2 | Center frequency | |
3 | Bandwidth | |
4 | Pulsewidth | |
5 | Antenna gain based on beamwidths in the azimuth and elevation planes | |
6 | Pulse repetition frequency | |
7 | Power aperture product (where is the subarray aperture size) | |
8 | Intrapulse modulation/pulse compression | |
9 | Waveform type | |
10 | Polarization | |
11 | Radar platform position | (,, ) |
12 | Radar platform velocity |
Serial # | Anomalies |
---|---|
1 | Targets not correctly detected (because targets are not resolved in range and angular domains). |
2 | Targets detected with SNR below a threshold (either due to clutter or RF interference). |
3 | Velocity variations above a threshold. |
4 | RCS fluctuations above a threshold. |
5 | Active RF emissions from malicious UAVs. |
6 | Classification of targets changes (due to the above listed anomalies). |
Serial # | Action | Action Label |
---|---|---|
1 | Change of center frequency | |
2 | Change of pulsewidth | |
3 | Change of bandwith | |
4 | Change of PRF | |
5 | Introduce intrapulse modulation or change of type of intrapulse modulation | |
6 | Change of number of pulses | |
7 | Change in transmit power | |
8 | Change of antenna beamwidth | |
9 | Change in the polarization of the phased array | |
10 | Change in type of sounding signal | |
11 | Changes in the beam scheduling | |
12 | Change of position of the radar platform onboard UAV |
Serial # | Radar Parameters | Parameter Values |
---|---|---|
1 | : | W : W |
2 | : | GHz : GHz |
3 | : | Hz : Hz |
4 | , | |
5 | : | |
6 | Number of pulses | |
7 | Noise radar waveform | |
8 | Vertical | |
9 | Radar noise figure | 5 dB |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Khawaja, W.; Yaqoob, Q.; Guvenc, I. RL-Based Detection, Tracking, and Classification of Malicious UAV Swarms through Airborne Cognitive Multibeam Multifunction Phased Array Radar. Drones 2023, 7, 470. https://doi.org/10.3390/drones7070470
Khawaja W, Yaqoob Q, Guvenc I. RL-Based Detection, Tracking, and Classification of Malicious UAV Swarms through Airborne Cognitive Multibeam Multifunction Phased Array Radar. Drones. 2023; 7(7):470. https://doi.org/10.3390/drones7070470
Chicago/Turabian StyleKhawaja, Wahab, Qasim Yaqoob, and Ismail Guvenc. 2023. "RL-Based Detection, Tracking, and Classification of Malicious UAV Swarms through Airborne Cognitive Multibeam Multifunction Phased Array Radar" Drones 7, no. 7: 470. https://doi.org/10.3390/drones7070470