An Adaptive Adversarial Patch-Generating Algorithm for Defending against the Intelligent Low, Slow, and Small Target

Abstract

The “low, slow, and small” target (LSST) poses a significant threat to the military ground unit. It is hard to defend against due to its invisibility to numerous detecting devices. With the onboard deep learning-based object detection methods, the intelligent LSST (ILSST) can find and detect the ground unit autonomously in a denied environment. This paper proposes an adversarial patch-based defending method to blind the ILSST by attacking its onboard object detection network. First, an adversarial influence score was established to indicate the influence of the adversarial noise on the objects. Then, based on this score, we used the least squares algorithm and Bisectional search methods to search the patch’s optimal coordinates and size. Using the optimal coordinates and size, an adaptive patch-generating network was constructed to automatically generate patches on ground units and hide the ground units from the deep learning-based object detection network. To evaluate the efficiency of our algorithm, a new LSST view dataset was collected, and extensive attacking experiments are carried out on this dataset. The results demonstrate that our algorithm can effectively attack the object detection networks, is better than state-of-the-art adversarial patch-generating algorithms in hiding the ground units from the object detection networks, and has high transferability among the object detection networks.

1. Introduction

The “low, slow, and small” target (LSST) is the abbreviation of the low-altitude, slow-speed, and small target. It refers to the aerial target with a flying height of less than 1000 m, a flight speed of less than 200 km/h, and a radar reflection cross-sectional of less than 2 m² [1]. They can be divided into multi-rotor, fixed-wing, and small unmanned helicopters. Along with the geometric growth of the drone market, the frequency of the incidents such as “black flight,” border cross, and drone attacks is increasing [2]. Moreover, if these drones are equipped with an intelligent system, they can operate independently in the denied environments [3,4].

Currently, the countermeasures against LSST are mainly based on kinetic and non-kinetic countermeasures.

The kinetic countermeasures mainly utilize small arms to destroy, interceptors to intercept, or aerial nets to capture LSST. All of these countermeasures depend on LSST detection. The detection of LSST is mainly based on radar [5,6], audio [7,8], video [9,10], and radio frequency [11,12].

Radar: The micro-doppler signal received using the doppler radar can help to identify the LSST [13]. However, because the LSST has a low radar cross-section, the detection of LSST using the radar faces significant challenges.

Audio: Analyzing the sound waves generated by LSST’s rotor can detect the LSST [14]. Although this can realize 24/7 uninterrupted detection, its detection range is not satisfiable and vulnerable to background noise.

Video: Computer vision technology enabled LSST detection [15,16] based on video images and object detection algorithms. However, it has a limited detection range and is vulnerable to rainy, foggy, and dusty weather.

Radio frequency: The LSST usually uses some special frequency band to communicate with their controllers [17]. By using a radio frequency scanning technic, one can monitor the LSST. However, in a real-world environment, a lot of radio frequency signals will lead to a high false alarm rate, making this technology unreliable on LSST detection tasks.

The non-kinetic countermeasures mainly use some interrupting technology to disturb the LSST. It mainly disrupts the control links to make the LSST out of control [18] or sends a false GPS signal [19] to make the LSST deviate from its route. However, intelligent LSST (ILSST) navigation mainly depends on a visual navigation system that does not rely on a GPS signal to navigate itself. It can operate on its own and barely require remote controls. Therefore, these LSSTs equipped with intelligent systems are nearly free from these kinds of disturbances.

ILSSTs are usually equipped with an object detection algorithm to locate ground units. To defend against the ILSST, we propose an adaptive adversarial patch-generating algorithm to make the ground unit invisible to the ILSST’s object detection algorithm. Due to its low speed, the ILSST can endure the latency of the object detection network. In exchange, it will achieve high detection precision. However, to guarantee real-time operation ability, the complexity of the network cannot be too high. Currently, few object detection networks can meet the requirement. Among them, due to its high precision, low complexity, and whole package of the solution to deploy on an embedded system, the YOLOv5 [20] algorithm is an optimal solution to use on the ILSST.

Therefore, this paper proposes an algorithm to attack the YOLOv5 network, which can hide the ground units from YOLOv5 and has high transferability among the other object detection networks. To our knowledge, this is the first time the adversarial patch has been used to defend against ILSST, which can protect critical ground units or facilities from being discovered or attacked by ILSST.

The overall structure of our proposed algorithm is shown in Figure 1. Our proposed algorithm in this paper consists of two parts: (1) the dataset labeling part provides the optimal coordinate and the size of the adversarial patches to the training dataset, as shown in the upper part of Figure 1; (2) the adaptive adversarial patch-generating network uses the labeled training dataset to construct a patch-generating model as shown in the lower part of Figure 1. Our contributions are as follows:

• We propose a novel idea for defending against the ILSST. Using the adversarial patch to defend against the ILSST can hide the ground unit from ILSST’s onboard object detection network.
• We design a patch-generating network that can generate patches on the optimal location of the object with optimal size without the ground truth of the objects.
• We propose a novel patch coordinate labeling method that fits a curve to an object using a set of sampling points and use this curve to find the point that affects the detection results most.
• We use the Bisectional search method to calculate the optimal size of the patch on objects, which enable our algorithm to generate different sized patch on different objects and increase the efficiency of the patch-generating algorithm.

Our paper is arranged as follows. In Section 2, we describe some of the related work on adversarial attacks and provide a detailed description of our algorithm. In Section 3, we experimentally evaluate our algorithm. Finally, we give the discussion and conclusion in Section 4.

2. Materials and Methods

2.1. Related Work

Recently, adversarial attacks have become an important concern in the field of computer vision and pose a significant threat to the remote sensing field as well. Consequently, adversarial attacks in remote sensing have become a growing concern in recent years [21]. Researchers have proposed various methods to generate adversarial examples that can deceive deep neural networks in remote sensing applications. Firstly, we summarize the adversarial attack algorithms, including adversarial perturbations and adversarial patches in classification and object detection tasks. Secondly, we review the application of adversarial attacks in remote sensing.

Adversarial perturbation: This type of attack misleads the network to give a wrong prediction by adding the imperceptible micro-noise to a normal sample. For the classification, there is much research. Szegedy [22] first discovered the adversarial phenomenon on image classification tasks and proposed an adversarial sample generating method L-BFGS. Based on this research, the fast gradient sign method (FGSM) has been proposed by Goodfellow [23]. FGSM adds perturbation along with the direction of the gradient. Its simplicity and efficiency have stimulated several pieces of research based on it [24,25]. These methods did not have much difference from the original FGSM algorithm. DeepFool algorithm [26] generates noise by comparing the distance between the points in sample spaces and the classification boundaries. The Jacobian-based saliency map attacks (JSMA) algorithm [27] calculates the salient scores on inputting images and adds perturbations according to the importance of the pixel to the output result, which can deceive the classification algorithm by merely changing a few pixels. C&W algorithm [28] uses the improved norms as the loss function to optimize the perturbation. Afterward, Rony et al. [29] improve the C&W algorithm to increase adversarial ability while still obtaining samples that approximate the visual perceptibility of the original algorithm. Croce et al. [30] proposed two algorithms, the auto-projected gradient descent, and the AutoAttacks, based on analysis of the suboptimal solutions phenomenon. Xiao et al. [31] proposes a generative adversarial network (GAN)-based method to efficiently generate stronger and more diverse examples for deep neural networks. Bai et al. [32] leverage both gradient-based and gradient-free optimization techniques to generate adversarial examples that are effective and transferable across multiple deep-learning models. Liu et al. [32] highly considers human perception and improves the inconspicuousness and transferability of generated patches. These described algorithms are typical classification attack algorithms; in this work, we concentrate on attacking the object detection algorithms.

For object detection algorithms, the research are not as much for attacking classification algorithms. Xie et al. [33] proposed an adversarial sample generation method for object detection algorithms. They proposed a dense adversarial generation algorithm that considers all objects simultaneously and optimizes the overall loss function to increase the transferability between object detection models. To attack a region proposal-based object detection algorithm, Li et al. [34] proposed the Robust Adversarial Perturbation (RAP). They construct two loss functions, one for the label and the other one for the shape. They implement an attack on region proposal-based object detection algorithms by optimizing these two loss functions. In order to speed up the attacking method and deal with the problem that the method used to attack region proposal-based object detection algorithm cannot attack regression-based object detection algorithm, Wei et al. [35] proposed the Unified and Efficient Adversary (UEA). The UEA is based on Generative Adversarial Network (GAN) [36] and combines high-level classification and low-level feature loss to jointly train an adversarial sample generator. Athalye et al. [37] generated three-dimensional adversarial samples to mislead the neural network by using the concept of “Expectation over Transformation” (EoT). Wang et al. [38] proposed a method that can attack the object detection algorithm that uses the Non-Maximum Suppression (NMS) method to filter out the redundant bounding box. Compressing the dimension of detection boxes, they paralyzed the NMS to make the detection result full of false positives. Although these object detection algorithms attacking methods can attack the object detection methods, they cannot be implemented in real-world circumstances because the perturbations added to the original image are too subtle to be captured by a camera.

Adversarial patch: This type of attack uses some specific-colored patches to replace an area in an image, which can mislead the network to give a false prediction. Currently, several breakthroughs have been made in the research of adversarial patches. Firstly, we summarize the adversarial patch-based attack for classification tasks. Brown et al. [39] propose adding a small patch to an image that can cause misclassification, even when the patch is small and visually imperceptible. In contrast, Karmon et al. [40] propose a localized and visible adversarial noise method to perturb only a small region of the image, making it more practical for real-world scenarios. Aran et al. [41] introduce the concept of using adversarial patches in QR codes, where the patch can be generated to be virtually invisible yet capable of causing misclassification of the QR code’s data. They extend the concept further [42], proposing a method that allows the attacker to control the QR code’s data and still generate an adversarial patch. Gittings et al. [43] propose a novel approach for generating adversarial examples that do not require any knowledge of the target model. It utilizes a deep image prior to synthesizing robust and visually realistic adversarial examples. Zhou et al. [44] introduce an approach that generates adversarial patches that are effective against a wide range of models, making it a practical choice for attackers who do not know the target model in advance. Bai et al. [45] propose an inconspicuous adversarial patch that can be added to any object and is effective against image recognition systems on mobile devices.

For the object detection tasks, Eykholt et al. [46] fooled the intelligent unmanned car’s detection system by sticking a small patch on a traffic sign. They applied a range of improvements to the Robust Physical Perturbation algorithm [47], introduced disappearance attack loss, and generated a range of small patches to attack the object detection algorithm. To deal with the issue that the adversarial patch cannot attack both the Faster RCNN [48] and YOLOv4 [49], Liu et al. proposed a DPATCH [50] algorithm. They iteratively train the adversarial patch that can attack the Faster RCNN and YOLOv4 object detection networks by simultaneously attacking the bounding box regression and object classification score. However, their work did not limit the pixel values that may cause the pixel value to exceed the valid range of an image. By introducing some augmentation to the patches, such as rotation, and changing the patch’s location and lighting condition, Lee et al. [51] improved the DPATCH algorithm. They used the Projected Gradient Descent (PGD) loss to confine the pixel values to a valid range. Thys et al. [52] applied the adversarial patch to the pedestrian detection system and successfully misled the pedestrian detector. They used the objectness and the classification score as part of the loss and the total variation loss and printable loss to make the patch applicable to the real world. Komkov et al. [53] attacked the best public face recognition system, ArcFace, by putting a sticker on the hat. They proposed a projection technique that projects the sticker to the image during the attack to make it “real-like”. Wang et al. [54] proposed an attacking patch-generating algorithm to make a specific class of objects invisible to the object detection algorithm. They analyzed the influence of the patch’s size on the object detection algorithm.

Adversarial attack in remote sensing: In 2018, Czaja et al. [55] discussed the challenges and potential impacts of adversarial attacks in remote sensing and provided an overview of the existing adversarial attack methods. Chen et al. [56] study the vulnerability of remote sensing image recognition systems to adversarial examples. They analyze the characteristics of adversarial examples in remote sensing images and propose a method to generate them using gradient-based optimization. They also propose a method [57] to generate universal adversarial examples that can attack multiple remote sensing models. Zhang et al. [58] analyze the universal adversarial patch attack for multi-scale objects in remote sensing images captured by unmanned aerial vehicles (UAVs). They introduce a scaling factor to make the adversarial patch valid for multi-scale objects. Bai et al. [59] extend the concept of universal adversarial examples to targeted attacks, where the generated adversarial examples are intended to fool a specific classifier. Shi et al. [60] investigate the impact of the adversarial attack on hyperspectral image classification using deep learning models. They evaluate the performance of several attacking algorithms on two hyperspectral datasets under different scenarios. Chen et al. [61] present an empirical study of the effects of adversarial attacks on different types of remote sensing data and evaluate the robustness of several popular deep neural network architectures. Li et al. [62] propose an approach to generate adversarial examples for synthetic aperture radar (SAR) image classification and evaluates the effectiveness of the generated adversarial examples on several CNN-based classifiers.

Although these adversarial generation algorithms are capable of deceiving the object detection algorithm, the adversarial patch they generate cannot simultaneously deal with the issue that (1) they cannot generate the patches on optimal coordinate on the object; (2) they cannot generate the patches with optimal size; (3) cannot patch on objects without ground truth; (4) cannot deal with objects that have a large range of scales, especially for small objects. The former three issues make it inefficient when deceiving the object detection algorithm. The last one makes the attacking method inefficient when defending against the LSST. Therefore, in this paper, we mainly focus on these four issues.

2.2. Method

To predict the optimal patch on objects without ground truth and make the patch-generating algorithm adaptive to different scaled objects, we designed a brand-new network (adaptive adversarial patch-generating network, AAPGNet) partially based on the YOLOv5 network structure, as shown in the lower part of Figure 1. The network extracts features from three scales and fuses them to produce patches on objects by outputting a specific-colored patch image and a patch mask that integrates the coordinates and size of the patches. The construction details of the network in the lower part of Figure 1 are described in Section 2.2.1.

The AAPGNet regresses the patch’s coordinate by regressing the offset of the patch relative to the object’s coordinate and the patch’s size by regressing the size of the patch relative to the object. To lead the network to learn the patch’s offset and size, we must provide the ground truth about these two values. Therefore, we need to label the objects with information about the patching coordinate and size of the patch on objects, as shown in the upper part of Figure 1. This labeling process will be described in detail in Section 2.2.2.

2.2.1. Adversarial Patch-Generating Network

In this section, we construct the AAPGNet to generate patches on the object without the ground truth. The AAPGNet consists of three parts, the backbone, the adversarial patching image generator, and the patching information generator, as shown in Figure 2. The patching information includes the patch’s coordinates on the patching image and the size of the patch. After obtaining the patching image and the patching information, we used the patching information to produce a mask image with a size equal to the patching image. At last, we can use this mask image on the patching image to select the patches that can directly patch onto the clean image.

To make the network have scale adaptiveness, we fused three different scales of feature information in the patch coordinate generator (as YOLOv5) and patching image generator. By doing this, the network can have fine-grained information in shallow layers and semantic information in deep layers when predicting the patch.

To reduce the feature map size of the network, we introduced the DeFOCUS operation. The DeFOCUS operation is the reverse process of the FOCUS operation in YOLOv5. It can reduce the feature map size to half, accelerating the network. The process of the DeFOCUS is shown in Figure 3.

After building the network structure, we must formulate a training goal as a loss function to train the network approach to our purpose. For our network, the loss function consists of two parts, (1) the adversarial loss function (ALoF) and (2) the patching information loss function (PILoF). The ALoF leads the detection algorithm to miss the target, and the PILoF leads the patching information generator to regress the patch’s coordinate and size.

• ALoF

To hide the objects from the object detection network, we lead the network to predict the objects as background. For this purpose, we used the binary cross-entropy function as the loss function.

$$los{s}_A=\frac{1}{n}{\displaystyle \sum _{i=0}^n-\left[p_i\log \left({\widehat{p}}_i\right)+\left(1-p_i\right)\log \left(1-{\widehat{p}}_i\right)\right]}$$

(1)

where the $p_i$ is the ground truth, the ${\widehat{p}}_i$ is the objectness the bounding box contains, and n is the number of the bounding boxes the object detection algorithm output. To lead the network to ignore the objects, we set the ground truth to zero in Equation (1), leading the network to regard the objects as the background. Thus, the ALoF is designed as follows:

$$los{s}_A=\frac{1}{n}{\displaystyle \sum _{i=0}^n-\log \left(1-{\widehat{p}}_i\right)}$$

(2)

2.

PILoF

The patch’s coordinate and size are predicted as an offset of the object’s center coordinate and the ratio of the object’s size, respectively. Assume that the predicted object’s center coordinate is $\left(\widehat{x},\widehat{y}\right)$, the size is $\left(\widehat{w},\widehat{h}\right)$, and the offset of the patch relative to the object’s center point is $\left({\widehat{r}}_x,{\widehat{r}}_y\right)$, the size of the patch relative to the object is $\widehat{k}$ (between 0 and 1). Then the patch’s coordinate will be

$${\widehat{x}}_p=\widehat{x}+\left[\left({\widehat{r}}_x-0.5\right)\times \widehat{w}\right]$$

(3)

$${\widehat{y}}_p=\widehat{y}+\left[\left({\widehat{r}}_y-0.5\right)\times \widehat{h}\right]$$

(4)

The size of the patch will be

$$\left({\widehat{w}}_p,{\widehat{h}}_p\right)=\left(\widehat{w}\times \widehat{k},\widehat{h}\times \widehat{k}\right)$$

(5)

Therefore, when designing the PILoF, we must also consider the object’s coordinates and size. For the object’s coordinate and size, we directly used the CIoU [63] as a loss. To predict a patch on the object, we additionally predict three parameters ${\widehat{r}}_x$, ${\widehat{r}}_y$, and $\widehat{k}$. We used Mean Square Error (MSE) for these three parameters as a loss function. At last, the PILoF is designed as follows:

$$los{s}_P=\frac{1}{\left|{\mathit{B}}_A\right|}{\displaystyle \sum _{i\in {\mathit{B}}_A}\left\{\begin{array}{l}\left(1-CIo{U}^i\right)+{\left[{\widehat{r}}_x^i-\left(r_x^i+0.5\right)\right]}^2\\ +{\left[{\widehat{r}}_y^i-\left(r_y^i+0.5\right)\right]}^2+{\left({\widehat{k}}^i-k^i\right)}^2\end{array}\right\}}$$

(6)

where B_A is the set of boxes that have been assigned labels using the YOLOV5’s label assignment method, $|·|$ is the length of a set, the $r_x$, $r_y$, and $k$ are the ground truths of the parameters ${\widehat{r}}_x$, ${\widehat{r}}_y$, and $\widehat{k}$, respectively, which will be given in Section 2.2.2.

Finally, the training loss will be as follows:

$$loss=los{s}_A+los{s}_P$$

(7)

To be noted is that there is an added or subbed 0.5 in Equations (3), (4) and (6). That is because when we predicted the parameters, we used the sigmoid activation function before the final output, which makes the value of ${\widehat{r}}_x$ and ${\widehat{r}}_y$ between 1 and 0. However, the ground truth of ${\widehat{r}}_x$ and ${\widehat{r}}_y$ is relative to the object’s center point (given in Section The Optimal Size of the Patches), which makes the $r_x$ and $r_y$ between −0.5 and 0.5.

2.2.2. Label the Objects in the Training Dataset with Optimal Patch Coordinates and Size

The constructed network in the former section needs the ground truth of the patch’s coordinates and size to train. The patch’s coordinate is predicted as the offset of the patch relative to the object’s center point, and the size is predicted as the size of the patch relative to the object. Therefore, in this section, we will describe the search process (as shown in the upper part of Figure 1) for the offset of the patch relative to the object’s center point and the size of the patch relative to the object.

The Optimal Coordinate of Patches

Varying positions of an adversarial patch on an object have different consequences on the object detection network. To locate the optimal coordinate of a patch on the object, we establish a score $S$ to quantify the influence of the patch. A YOLO-type object detection network usually predicts a bunch of bounding boxes $\mathit{B}=\left\{b_1,b_2,\cdots ,b_n\right\}$ for an image I. For an object $o_j$, only a few parts of the bounding boxes in B can be regarded as valid boxes, and the others are not counted. Therefore, we only used these valid boxes to calculate the score S for an object $o_j$. Here, the valid boxes represent the boxes that have proper Intersection over Union (IoU) with the object $o_j$, and the predicted class is consistent with the object $o_j$. Thus, we used the IoU to select the valid boxes. Specifically, we consider the bounding boxes to be valid when it has a greater than 50% IoU with the object $o_j$, and the predicted class is consistent with the object $o_j$. Finally, the score S is calculated for an object $o_j$ as follows:

$$S_{o_j}=\frac{1}{\left|{\mathit{B}}_v\right|}{\displaystyle \sum _{b_i\in {\mathit{B}}_v}{D}_{obj}^{b_i}\left(o_j\right)}$$

(8)

where ${\mathit{B}}_v$ is the valid box set for the object $o_j$, and $D_{obj}^{b_i}\left(o_j\right)$ is the objectness probability of the box b_i given on the object $o_j$.

We aim to find the patching point that makes the object detection network give the minimum S to the patched object. For this purpose, we sampled several points on the object and used these points to fit a surface on the object through the least-squares algorithm.

We first used a GAN to train an adversarial noise-generating network to sample points on an object. This adversarial noise-generating network’s structure is similar to the network in Figure 2, but without the patching information generator, as shown in Figure 4. For constraint of the noise mainly on the object, we set the constraint function as follows:

$$los{s}_c=\frac{1}{\left|O\right|}{\displaystyle \sum _{o_j\in O}\frac{1}{\left|T_j\right|}{L}_1\left(T_j\right)}$$

(9)

The $L_1(·)$ is the $L_1$ normalization, $T_j=\left\{pix|pix\in o_j\right\}$ is the set of pixels that belong to the object $o_j$ in adversarial noise image, and $O=\left\{o_j|o_j\in I\right\}$ is the set of objects that belong to the image I.

The training of the adversarial noise-generating network also requires adversarial loss, which is identical to the adversarial loss designed for the adversarial patch-generating network in Section 2.2.1, i.e., the $los{s}_A$. Therefore, the final loss for the adversarial noise-generating network is as follows:

$$loss=los{s}_A+los{s}_c$$

(10)

After training the GAN, we set a noise-through window whose size is t (smaller than 1) times the width and height of each object. With the target width and length as one and the object center point as the origin, slide the noise window on the object surface in the x and y directions in steps of 0.1. Finally, we can collect a set of sampling points, as shown by the colored dots in the upper part of Figure 1.

$$\left\{r_x,r_y,s_{y,x}\right\}=\left\{\begin{array}{cccc}\left(r_1,r_1,s_{1,1}\right)& \left(r_2,r_1,s_{1,2}\right)& \cdots & \left(r_9,r_1,s_{1,9}\right)\\ \left(r_1,r_2,s_{2,1}\right)& \left(r_1,r_3,s_{2,2}\right)& \cdots & \left(r_9,r_1,s_{2,9}\right)\\ ⋮& ⋮& \cdots & ⋮\\ \left(r_1,r_9,s_{9,1}\right)& \left(r_2,r_9,s_{9,2}\right)& \cdots & \left(r_9,r_9,s_{9,9}\right)\end{array}\right\}$$

(11)

where $r_i=i/10-0.5$ represents the offset of the window relative to the object’s center point, $s_{i,j}$ is the score S calculated under the ($r_x$, $r_y$) step of the noise-through window.

Based on the sampled points, we used the following equation to formulate the curve surface on the object, as shown by the colored surface in the upper part of Figure 1.

$$S=\frac{1}{2}(r_x,r_y)\mathit{A}\left(\begin{array}{c}{r}_x\\ r_y\end{array}\right)+\left(r_x,r_y\right){\mathit{W}}^T+b$$

(12)

$$\mathit{A}=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{12}& a_{22}\end{array}\right],\mathit{W}=\left({\omega }_1,{\omega }_2\right)$$

(13)

Equation (12) can be transformed to

$$\begin{array}{l}S=\left[r_x^2,r_y^2,r_x{r}_y,r_x,r_y,1\right]{\left[\frac{1}{2}{a}_{11},\frac{1}{2}{a}_{22},a_{12},{\omega }_1,{\omega }_2,b\right]}^T\\ =:{\mathit{R}}^T\mathsf{\beta }\end{array}$$

(14)

The least-squares equation will be

$$\underset{\beta }{\min }\frac{1}{2}\Vert S-{\mathit{R}}^T\beta \Vert$$

(15)

The solution to this function will be

$$\beta ={\left({\mathit{R}}^T\mathit{R}\right)}^{+}{\mathit{R}}^{T}S$$

(16)

The ${(·)}^{+}$ indicates the Moore–Penrose generalized inverse. Finally, we can solve the optimization issues Equation (17) to obtain the minimum point of S as the optimal coordinate of the patch.

$$\underset{r_x,r_y}{\min }\frac{1}{2}\left(r_x,r_y\right)\mathit{A}\left(\begin{array}{c}{r}_x\\ r_y\end{array}\right)+{\mathit{W}}^T\left(\begin{array}{c}{r}_x\\ r_y\end{array}\right)+b$$

(17)

which is approximately

$$\left(r_x,r_y\right)={\mathit{A}}^{-1}{\mathit{W}}^T$$

(18)

The inverse of A does not always exist because, on some objects, the surface is not precisely fit Equation (12). For these objects, we directly pick out the smallest point in the point set $\left\{r_x,r_y\right\}$ as the optimal coordination of the patch.

The Optimal Size of the Patches

To determine the optimal patch size, we trained a patching image-generating network using the network structure in Figure 4 with the $kw\times kh$ sized patch on the object’s optimal patching point (as labeled in Section The Optimal Coordinate of Patches). The w and h are the width and height of the object, respectively. When training the network, (1) we used the network to generate a patching image with a size equal to the input image, (2) uncovered $kw\times kh$ sized patches from the patching image on the point corresponding to the object’s optimal patching point, (3) patch these patches to the object’s optimal patching point, and (4) fed this patched image to the object detection network. The loss we used in this training process is $los{s}_A$.

The larger the size, the easier it is to fool the object detection network [54]. However, if k is too large, the patch will lose its adversarial ability (this will be experimentally proven in Section 3.6). Therefore, the k needs to be considered comprehensively to ensure that (1) the patch is not too large to weaken the adversarial ability of the patch and (2) the patch is not too small to deceive the algorithm.

In this paper, we chose three k values according to the algorithm’s performance degradation on the training dataset, which also divided the object in training datasets into three groups, i.e., the easy to fool, the general to fool, and the hard to fool. Expressly, we set three performance degradation thresholds f₁ < f₂ < f₃, and we choose three k values ($k_1$, $k_2$, $k_3$) for three thresholds. The principle to choosing $k_i$ value is to train $k_{i}w\times k_{i}h$ sized patches on the objects to make the performance degradation approximately (±10%) equal to each threshold (from low to high) and remove the former undetected objects from the dataset for the latter $k_i$ value. By doing this, we divide all the objects in the training dataset into three sets $O_{k_1}$, $O_{k_2}$, and $O_{k_3}$. The $O_{k_i}$ contains the object that the detection network cannot detect under a $k_{i}w\times k_{i}h$ sized patch. The three sets of objects are incompatible because we removed the undetected objects from the dataset for the latter $k_i$ value.

In the end, we used the Bisection method to search the optimal patch size on each set of objects, as shown in Algorithm 1. We set a threshold score of ts as a stopping point to use the Bisection method.

In Algorithm 1, the value $\epsilon$ acts as a relaxation variable, and the values l and h are the upper and lower bounds of the search region of the Bisection method. The $S_{o_j}^{P_{new}}$ represents the score S under the patch $P_{new}$ for the object $o_j$, which is calculated as follows:

$$S_{o_j}^{P_{new}}=\frac{1}{\left|{\mathit{B}}_v\right|}{\displaystyle \sum _{b_i\in {\mathit{B}}_v}{D}_{obj}^{b_i}\left(o_j|P_{new}\right)}$$

(19)

The $D_{obj}^{b_i}\left(o_j|P_{new}\right)$ is the objectness probability of the box b_i given on the object $o_j$ under the condition of patching a patch $P_{new}$ on the object $o_j$.

After running this algorithm on three sets of objects, the algorithm will label all the objects in the training dataset.

Algorithm 1: Patch size calculation algorithm

Preparation 1: set a thresh score of ts, a relaxation value $\epsilon$, $l=0$, $h=k_i$ 2: train a GAN with the $k_{i}w\times k_{i}h$ sized patch on the object’s optimal patching point. 3: use GAN to produce a $k_{i}w\times k_{i}h$ sized patch $P_{org}$ for every object in $O_i$ Start 1: for every object $o_j$ in $O_i$: 2:  while $\left|S_{o_j}^{P_{new}}-ts\right|>\epsilon$: 3:   let $k=\frac{\left(l+h\right)}{2}$ 4:   cut the $kw\times kh$ area from the center of the $P_{org}$ as patch $P_{new}$ 5:   if $S_{o_j}^{P_{new}}<ts$: 6:    let $l=k$ 7:   else: 8:    let $h=k$ 9:  save the k as the optimal patch size of the object $o_j$ end

3. Results

3.1. Dataset

In this research, we used the Digital Combat Simulator (DCS) to simulate the UAV view environment and collect images from it as a dataset. There are many realistic military target modules in DCS, including aircraft, tanks, ships, etc. The mission editor in Digital Combat Simulator (DCS) is a powerful tool that allows users to create custom missions and combat scenarios. It enables the user to define mission objectives, such as destroying enemy targets, performing air support missions, and escorting targets.

To collect the data, we used the DCS mission editor to set different target marching tasks in various scenarios and simulated aerial observations of each target. The targets include tanks, armored vehicles, warships, and carriers, and the background includes the sea surface, port, wilderness, town, etc. During the observation, we set the altitude of the UAV from about 20 m to 1000 m to simulate the LSST’s flying height. The view is slowly zoomed in to simulate the scenario of an LSST approaching the target. The resolution of the data image is 1920 × 1080, and when scaled down to the network input size (640 × 640), the target size ranges from a minimum of 11 × 11 to a maximum of 304 × 304. When collecting the data, we set the altitude of the UAV from about 20 m to 1000 m to simulate the LSST’s flying height. We collected 9150 images and split them into training (90%) and testing (10%) datasets.

3.2. Implementation Details

We implemented the YOLOv5 [20] network for the detection network according to the official YOLOv5 version 6. Based on this work, we implemented our attack algorithm on YOLOv5. We also implemented the YOLOv3 [64], YOLOv4 [49], YOLOX [65], and Faster RCNN [48] algorithms to test the transferability of our attacking algorithm.

During the labeling of the patch’s coordinate on the object, we set noise-through window size t to 0.3, set the $f_1$, $f_2$, $f_3$ to 30%, 60%, 90%, and accordingly set the $k_1$, $k_2$, $k_3$, ts to 0.17, 0.26, 0.42, 0.1 when we are labeling the optimal size of the patches, respectively. We also added the total variation loss [66] to control the variation of the patch colors. To improve the algorithm’s robustness to lighting and object position variations, we applied Gamma transfer, random flipping (in all four directions), and random rotations ranging from −5° to 5° on patched images during the network training. We avoided larger rotations during training as they may result in inaccuracies in the ground-truth bounding boxes.

This paper used the adversarial patch to hide the object from the object detection network. Therefore, to evaluate the algorithm’s performance, we introduced an evaluation index named average precision drop (APD) to evaluate the performance of the attacking algorithm. We define the APD as the drooping rate of the detection algorithm’s Mean Average Precision (mAP). Accordingly, the APD is calculated as follows,

$$\mathrm{APD}=\frac{\mathrm{mA}{\mathrm{P}}_b-\mathrm{mA}{\mathrm{P}}_a}{\mathrm{mA}{\mathrm{P}}_b}$$

(20)

The mAP_b and mAP_a are the mAP of the algorithm before and after the attack, respectively. mAP is used instead of the recall rate for APD because the recall rate depends on the threshold of the objectness score used to select the object from the background. Thus, the recall rate is not as stable as mAP.

To evaluate the performance of our attacking algorithm, we first tested all the object detection networks on our dataset. The results are shown in Figure 5. To evaluate the performance of our attacking algorithm, we first tested all the object detection networks on our dataset. The results are shown in Figure 5.

All the algorithms are programmed based on Pytorch 1.10 and carried out on TITAN RTX GPU.

3.3. Evaluation of the Scale Adaptiveness

To evaluate the efficiency of the strategy used to increase the scale adaptiveness of our algorithm, we constructed four networks: (1) the network AAPGNet drawn in Figure 2, (2) the network without branches 2 and 3 named AAPGNet1, (3) the network without branch 3 named AAPGNet12, (4) the network without branch 2 named AAPGNet13. We also split the objects in the test dataset into two groups: (1) the objects with a size smaller than 50 × 50 pixels, and (2) the objects with a size bigger than 50 × 50 pixels. We tested these four networks on the testing dataset and separately counted the detection results on these two groups of objects. The results are listed in

Table 1. Testing result of the scale adaptiveness.

Dataset	mAPb	Attack Algorithms	mAPa	APD
Small objects	74.11%	AAPGNet	21.57%	70.90%
		AAPGNet1	40.99%	44.69%
		AAPGNet12	29.28%	60.49%
		AAPGNet13	24.60%	66.80%
Large objects	89.57%	AAPGNet	13.74%	84.66%
		AAPGNet1	34.58%	61.39%
		AAPGNet12	24.22%	72.96%
		AAPGNet13	25.72%	71.29%

.

Table 1 shows that the network with three branches has the highest APD, and the network with only one branch has the lowest APD. The gap is more prominent on small objects, which means that the network with three branches has more adaptiveness to small-sized objects. That is because of the down-sampling operation, and the deep network structure makes the network lose fine-grained information about the objects, making the network ignore small objects. The three-branched network fuses different scaled feature information to obtain semantic and fine-grained information about the objects. However, the other three networks do not have sufficient fine-grained information about the objects.

3.4. Evaluation of the Optimal Coordinate

We construct five training datasets labeled with five different patch coordinates on objects to verify the optimal patch coordinate. The five datasets are identical except for the coordinate of the patches. In these five training datasets, only one is labeled with optimal coordinates using the method described in this paper. The others are all labeled with random coordinates on objects. After constructing the training datasets, we construct five patch-generating networks on each of these five datasets using the network in Figure 2. We tested these five networks on our testing dataset and presented the results in Figure 6.

Figure 6 shows that the network trained on the dataset with optimal patch coordinates has the highest ADP, demonstrating that the optimal coordinates of patches are much better than random coordinates. This is because we are labeling the objects with patch coordinates that affect the objectness most, and detecting results of the object detection network mainly rely on the objectness.

3.5. The Influence of Different Patch Shapes

The shape of patches can affect the performance of the patch. To verify this, we generated adversarial patches with various shapes for the testing dataset, including rectangles, squares, ellipses, and circles. In the case of the ellipse, we set two axes as h/2 and w/2. For the square, we set the length of the side as k × min(h, w), and for the circle, we set the radius as k × 1/2 min(h, w). It is worth noting that when searching for the optimal patch coordinates (or optimal patch size), the shape of the noise-through window (or patch) needs to be adjusted to the corresponding shape. We recorded the results in

Table 2. Comparison of the different-shaped patches.

Shape	mAPb	mAPa	APD	Average Size
Rectangle	81.84%	17.8%	78.25%	0.32
Square		15.86%	80.62%	0.41
Elliptical		18.49%	77.4%	0.27
Circle		16.43%	79.92%	0.35

.

Table 2 indicates that the highest APD is achieved when the patch shape is square, but the patch size is larger than other shapes and may exceed the target boundary. In terms of patch size, the elliptical patch has the smallest size, but the success rate of APD is the lowest, although it is only slightly different from that of the rectangular patch. If we use “APD/patch size” as the measure of patch deception efficiency, then the order of efficiency for several shapes is elliptical > rectangular > circle > square.

3.6. The Relationship between Patch Size and the Patch’s Adversarial Ability

In Section The Optimal Size of the Patches, we mentioned that if the patch size is too large, it will lose the adversarial ability. Generally, the bigger the patch’s size, the weaker the adversarial ability. To prove this conclusion, we designed an experiment. We used the GAN network structure drawn in Figure 4 to train two patching image-generating models. The first one was trained with $0.6w\times 0.6h$ sized patches on the object’s center point, named 0.6p-net. The second one was trained with $0.4w\times 0.4h$ sized patches on the object’s center point, named 0.4p-net. Afterward, (1) we patched each object in the test dataset with a $0.4w\times 0.4h$ ($0.6w\times 0.6h$) sized patch using the patching image the 0.6p-net (0.4p-net) generated. The w and h are the width and height of the object, respectively. We recorded the APD in

Table 3. Comparison of the different-sized patches.

Control Group	mAPb	mAPa	APD
0.6p-net with $0.6w\times 0.6h$ sized patch	81.84%	10.04%	87.73%
0.4p-net with $0.4w\times 0.4h$ sized patch	81.84%	30.62%	62.59%
0.6p-net with $0.4w\times 0.4h$ sized patch	81.84%	53.55%	34.57%
0.4p-net with $0.6w\times 0.6h$ sized patch	81.84%	28.36%	65.35%

.

In this experiment, when training the patching image-generating models, we uncovered corresponding sized patches, for example $0.4w\times 0.4h$, from the patching image on the point corresponding to the object’s center point from the patching image. Table 3 shows a significant decline in APD when decreasing the patch size of 0.6p-net but not a considerable increase when increasing the patch size of 0.4p-net. This phenomenon indicates that the patching image generated by 0.6p-net has a weaker adversarial ability than the one generated by 0.4p-net.

3.7. Evaluation of the Transferability

Our attacking algorithm is implemented on YOLOv5 object detection networks. We needed to test its attacking transferability to other object detection networks. Therefore, we chose Faster RCNN, YOLOv3, YOLOv4, YOLOv5, and YOLOX five networks to test the attacking transferability. During the test, we separately trained five networks on our training datasets and generate the adversarial patches on the test dataset only for YOLOv5. We used this patched testing dataset to attack four other object detection networks. The results are listed in

Table 4. Transferability testing results.

Algorithm	mAPb	mAPa	APD
YOLOv5	81.84%	17.8%	78.25%
Faster RCNN	74.59%	22.95%	69.23%
YOLOv3	68.52%	15.48%	77.41%
YOLOv4	79.06%	21.13%	73.27%
YOLOX	81.13%	23.5%	71.03%

.

Table 4 shows that our attacking algorithm can attack the YOLO type one-stage algorithm and the Faster RCNN two-stage algorithm with only one model. Our attacking algorithm is designed for YOLO type (one-stage algorithm) object detection network. However, it has transferability to the two-stage object detection network. That is because we trained our attacking model on the whole training dataset, which makes the model learn the common features of the training dataset. Therefore, if the object detection network is construed based on this training dataset, there will be a high probability that the object detection network can be attacked by the attack model trained on this dataset for other object detection networks.

3.8. Comparing with Other Attacking Algorithms

To compare our algorithm with other adversarial patch-generating algorithms, we implemented two other patch-generating algorithms, i.e., OBJ in paper [52] and the algorithm in paper [54] (for convenience, we named it PG in this paper). For fairness, we removed the printable loss in algorithm OBJ and PG during the comparison. We set the parameters a, $\alpha$, and $\beta$ in PG to 0.125, 0.6, and 0.4, respectively. We recorded the average size of the patches on all objects as the size of our algorithm in Figure 7, and set the patching size of the OBJ to the size of our algorithm’s average size. The testing results are shown in Figure 7.

In Figure 7, the size represents the proportion of the patch size to the object size. Our algorithm gets the highest ADP when the size is equal to the other two algorithms, indicating that our attacking algorithm is more efficient in hiding the object from ILSST.

There is something that should be noticed. In this test, the other two algorithms needed to provide the ground truth about objects when patching the patch on each object, but our algorithm does not need the ground truth about the objects. We also visualized some of the testing results in Figure 8. In the third column of Figure 8, some patches exceed the objects, which is hard to realize in a real-world situation. In the second column of Figure 8, several ground units failed to be hidden from the detector. However, in the first column of Figure 8, not only did the patch not exceed the objects, but it also deceived the detector, which indicates that our algorithm is more efficient when hiding the object from the detector.

4. Discussion

In this paper, we presented an adaptive adversarial patch generation algorithm designed to protect military ground units from ILSST attacks or reconnaissance. In the design process, we considered the target characteristics (multiple scales) of LSST and devised an adaptive adversarial patch-generating network. During the adversarial patch generation, the network extracted features from three scales. Based on these three scaled features, an adversarial image generator was employed to produce an adversarial image, while a patching information generator predicted the positions and sizes of the patch across the three scales. Finally, based on its position and size information, we located the corresponding patch in the adversarial image and patched it onto the target. To verify the effectiveness of the multi-scale feature extraction network, we conducted ablation experiments in the experimental section. The results confirmed the efficiency of our algorithm.

Previous adversarial patch-generating algorithms generated adversarial patches by manually specifying their position and size, which is inefficient in deceiving target detection algorithms. To deal with this issue, we used the least squares method and binary search to find the optimal patch position and size for the targets in the training dataset and trained the patch generation network using this position and size information. We conducted several experiments to verify the patch’s efficiency. The results demonstrate that our algorithm can generate optimal-sized adversarial patches for targets at different positions, greatly improving the adversarial efficiency of the patches.

There is something we noticed in our experiments. The patch-generating algorithm in this paper generates patches for multiple objects in the input image, and these objects can belong to different classes. Intuitively, the patches that hide the object from the object detection network should have different color patterns. However, we found that the color patterns of the patches are surprisingly similar. We speculated that it is because the object detection networks detect all the objects as one class, i.e., the objectness class, when it discerns the object from the background. The pass-through of the same class label is consistent, which led to the noise patterns that interfere with prediction results for the same object class being similar.

The adversarial patches are needed to be transferred to real-world military units. Currently, researchers are using the non-printable loss [66] when training the patch to transfer the electronic patch to the real world. The transformation also needs the patch to be smoothly colored. The smooth color process can be achieved by using the total variation loss to restrict the frequent abrupt change of colors which is already used in our paper (mentioned in Section 3.2). However, this process will significantly reduce the adversarial effectiveness of the patch, and further extensive studies are needed to address this issue.

5. Conclusions

In this paper, we proposed an ILSST defending strategy to hide the object from the ILSST’s onboard object detection networks. The patch-generating algorithm proposed in this paper can automatically generate patches on the objects with optimal coordinates and size without the ground truth. When constructing the patch-generating network, we also considered that the ground units have an extensive range of scales from the ILSST’s view. In the end, extensive experiments are applied, and the performance of the proposed algorithm is verified. The result shows that our algorithm can effectively generate patches on objects without ground truth, has high transferability to other object detection networks, and is more effective than other similar state-of-the-art algorithms.

Currently, we have mainly verified the feasibility of our proposed defending algorithm in the simulation environment. It requires further work to establish a real-world environment and transfer the patches to real-world ground units, which requires considering clouds, light conditions, etc. In future work, we will focus on establishing real-world environments and transferring the patches to real-world ground units.