Benchmarking Adversarial Patch Selection and Location

Abstract

Adversarial patch attacks threaten the reliability of modern vision models. We present PatchMap, the first spatially exhaustive benchmark of patch placement, built by evaluating over $1.5\times {10}^8$ forward passes on ImageNet validation images. PatchMap reveals systematic “hot-spots” where small patches (as little as 2% of the image) induce confident misclassifications and large drops in model confidence. To demonstrate its utility, we propose a simple segmentation-guided placement heuristic that leverages off-the-shelf masks to identify vulnerable regions without any gradient queries. Across five architectures-including adversarially trained ResNet-50-our method boosts attack success rates by 8–13 percentage points compared to random or fixed placements.

1. Introduction

1.1. Background

Deep neural networks have achieved near-human accuracy on many vision benchmarks, yet they remain alarmingly brittle to small, localized perturbations. In particular, adversarial patches printed stickers or posters that can be physically applied to a scene have been shown to reliably fool classifiers and detectors in both digital and real-world settings [1,2,3]. Their practical threat spans self-driving cars, surveillance systems, and biometric authentication.

A patch attack is defined not only by what the patch looks like, but also by where it is placed. While there has been extensive work on crafting patch appearance (optimizing texture, shape, and color) [1,2], far less attention has been paid to the equally critical question of patch placement. In practice, many defenses and evaluation protocols implicitly assume a fixed placement (e.g., center or corner of the image), or optimize location jointly with texture on a limited number of examples [4,5]. At the same time, standardized resources such as ImageNet-Patch [6] provide transferable patch textures and have accelerated reproducible evaluation of patch content, but they do not characterize spatial vulnerability across the full range of possible locations.

1.2. Problem Statement and Gap

Without a systematic, large-scale study of patch placement, it is difficult to answer basic questions that directly affect both attack evaluation and defense design: Which regions of natural images are consistently most susceptible to small patches? How does vulnerability change with patch size? And how reliable are placement strategies that aim to approximate worst-case locations without costly model queries? Existing work provides only sparse evidence. Either by evaluating a small set of handpicked placements or by optimizing locations per image with limited coverage [4,5]. This leaves a gap: we lack a publicly available, spatially exhaustive benchmark that maps vulnerability as a function of location under controlled patch content, enabling fair comparison of placement strategies and revealing dataset-level “hot-spots”.

1.3. Motivation

Consider two identical patches: one pasted over a salient object region, the other over background. Intuitively, the former will more strongly disrupt classification, yet existing benchmarks provide only limited support for this intuition because they do not exhaustively probe placement. A comprehensive spatial analysis would quantify where patches are most harmful, how often strong effects occur, and whether simple, model-agnostic cues can reliably identify vulnerable locations.

1.4. PatchMap

We introduce PatchMap, a spatially exhaustive benchmark for adversarial patch placement built using transferable patch textures from ImageNet-Patch [6]. PatchMap v1 focuses on two representative, high-impact patches (IDs 2 and 6) and evaluates three square sizes (50, 25, 10 px) at every stride-2 location on a 2000-image subset of the ImageNet-1K validation set. This yields $1.5\times {10}^8$ forward passes on a standard ResNet-50, recording both predicted labels and softmax confidences at each location. The resulting dense “vulnerability maps” uncover stable hot-spots, quantify confidence collapse patterns, and expose spatial motifs that are invisible to sparse evaluations. The evaluation method is explained at Figure 1.

Figure 1. PatchMap evaluation method: We apply patch for any location in the image, and for each location we estimate the predicted class of the model and the confidence of the model on the ground truth class (y).

1.5. Contributions

• PatchMap dataset. A public release of 100 M+ location-conditioned predictions at https://huggingface.co/datasets/PatchMap/PatchMap_v1 (accessed on 15 May 2025), scaling to 6.5 B entries.
• Rigorous analysis. Unified definitions of attack-success rate (ASR) and confidence drop ($\Delta \mathrm{conf}$), evaluated across location, patch size, and model robustness, exposing recurring spatial vulnerabilities.
• Segmentation-guided placement. A fast, zero-gradient heuristic that selects high-impact locations via off-the-shelf semantic masks, boosting ASR by 8–13 pp over random or fixed baselines-even on adversarially trained networks.

By decoupling what a patch looks like from where it lands, PatchMap lays the groundwork for location-aware defenses, adaptive attacks, and deeper insight into the spatial dynamics of adversarial vulnerability.

2. Related Work

2.1. Universal and Physical Adversarial Patches

Ref. [1] first introduced universal patches that consistently coerce a classifier into an attacker–chosen label. Because these patches are physically realizable, later studies printed them to fool real world systems: LaVAN places a small visible blob on the background [7], while RP2 stickers make stop signs read as speed limits [2]. Domain specific variants exist for face recognition, where perturbations are embedded in eyeglass frames [3], and for traffic sign detection [8]. The feasibility of such localized, robust attacks motivated datasets such as ImageNet-Patch [6], which supplies transferable patch textures later reused in our work.

2.2. Patch Placement and Location Optimisation

Early attacks fixed the patch at a predetermined corner or the object centre [2,7], implicitly assuming position mattered little. Subsequent research relaxed this by optimising location jointly with texture: LOAP employs gradient descent on both variables [4]; Simultaneous-Patch uses reinforcement learning (RL) to search position in a black-box setting [5]; PatchAttack learns an RL policy subject to a query budget [9]. Real-world “Adversarial Sticker” work further optimizes in-plane rotation and scale of printed patches [10]. Ref. [11] Examines patch placement selection on of street sign and the effect on object detection models. Generative Dynamic Patch Attack (GDPA) [12] a use GANs to propose both content and location, showing that placement is tightly coupled with texture generation. Attention-guided methods such as PS-GAN [13] infer high impact regions from saliency maps, while Shapeshifter extends the idea to object detection [2]. Despite this progress, prior studies either optimise on a handful of images or evaluate sparse locations; none offers an exhaustive, publicly released map of spatial vulnerability.

2.3. Context Awareness and Adaptive Attacks

Modern attacks adapt the patch to scene context. Distributed stickers cover multiple object parts for occlusion robustness [10], and Dynamic-Patch sequences adjust appearance over time [12]. Beyond classification, context-aware patches fool detectors, depth estimators, and segmentation networks, often by incorporating physical transforms (viewpoint, lighting) into training [2]. Several works exploit semantic priors: masking patches to background only, or using segmentation cues to stay off salient regions [14]. Our segmentation-guided heuristic takes the opposite tack-seeking the most segmentation-confident region-while remaining optimization-free and substantially faster than RL or gradient-based search. [15] explores leveraging of XAI such as Grad-CAM in order to select patch placement with an impact on model accuracy and that maintains difficulty to detect by the human eye.   PatchMap complements the above literature by offering the first large-scale, architecture-agnostic benchmark that decouples patch appearance from exhaustive placement, enabling quantitative comparison of any optimization-free or optimization-based strategy on common ground.

Adversarial attack methods are mostly compared using the ASR (Attack Success Rate) metric denoted as:

$$ASR={\displaystyle \frac{1}{N_{\mathrm{cc}}}}\sum _{i=1}^{N_{\mathrm{cc}}}\mathbb{I}\left[\widehat{y}\ne y_i\right],$$

(1)

where y is the originally predicted ground truth, and $\widehat{y}$ is the prediction after the appliance of the attack. $N_{\mathrm{cc}}$ is the number of correctly classified examples.

3. Dataset Design

3.1. Patch Source

We use the ten publicly released ImageNet-Patch adversaries of Li et al. [6]. Each $50\times 50$ RGB patch is gradient–optimised for a distinct target class and shown to transfer across common CNN backbones. Fixing this set guarantees strict reproducibility and avoids the confound of re-optimising patch content. The patches are applied without rotation.

3.2. Spatial Sweep

For every ImageNet-1K validation image ($224\times 224$), each patch is pasted on a dense stride-2 grid of feasible centres. The grid contains $112\times 112=\mathrm{12,544}$ positions and is evaluated at three square sizes: native $50\times 50$, down-scaled $25\times 25$, and $10\times 10$ px covering almost the whole image while keeping compute tractable. In total 50,000 images × 10 patches × 3 sizes × 12,544 locations $\approx 1.9\times {10}^{10}$ potential placements. By batching inference and discarding placements that would fall outside the frame, we actually run $\mathbf{1.1}\times {\mathbf{10}}^{\mathbf{8}}$ forward passes-roughly two orders of magnitude more than any previous location study.

3.3. Model Attacked

All placements are evaluated on a standard ResNet-50 [16] from torchvision, pretrained on ImageNet and not adversarially fine-tuned. Future PatchMap releases will incorporate additional backbones, but v1.0 deliberately fixes a single architecture to keep the file size manageable.

3.4. Recorded Data Format

PatchMap is sharded: every triple (image_id, patch_id, patch_size) is stored in its .npy file named {img}_{patch}_{size}.npz. Each file contains a single NumPy array of shape $2\times 112\times 112$: the first slice holds the predicted class indices (int16); the second holds the corresponding soft-max confidences (float32). With $50,000$ images, ten patches and three sizes, v1.0 comprises $1.5$ million files totalling ∼1.5 GB after compression. This fine-grained layout lets users download only the subsets they need and stream batches directly from disk without monolithic archives.

3.5. Choice of Parameters

In choosing the parameters we performed an ablation study over 50 random images. We performed the evaluation on a 1-d stride and over all patches to select an optimal stride and optimal patches for the larger dataset. We perfomed the ablation over patches of size $50\times 50$. We chose a 2-stride grid for our dataset in comparison to the original image, as it yields identical results as for the full 1-stride grid, saving a significant amount of computations with no compromise in accuracy, as shown in Figure 2. The reason for that phenomenon is the architecture of resnet-50 containing a stride of 2 on the first convolution of the model. We chose patches 2 (“Plate”) and 6 (“Guitar”) as they have yielded better mean optimal ASR (as measured in Table 1) than any other patch on the same ablation case.

Figure 2. Ablation of the MAE (Mean Absolute error) of the confidence per mask between the 1-stride mask an x-stride mask. Blue dots represent sample values.

Table 1. Ablation: mean optimal ASR: fraction of images for which some location causes a misclassification. Patches 2 and 6 yields better optimal location, allowing for a higher attack success rate on an optimal patch placement location. In bold are the top-2 highest $ASR$ patches.

Patch Size	Patch
Patch 0 (“Soap Dispenser”)	0.82
Patch 1 (“Cornet”)	0.78
Patch 2 (“Plate”)	0.84
Patch 3 (“Banana”)	0.78
Patch 4 (“Cup”)	0.78
Patch 5 (“Typewriter”)	0.83
Patch 6 (“Guitar”)	0.94
Patch 7 (“Hair Spray”)	0.80
Patch 8 (“Sock”)	0.76
Patch 9 (“Cellphone”)	0.76

3.6. Resources

All benchmark evaluations can be performed with a single processor with at least 8 GB of memory (with/without GPU).

3.7. Public Release

PatchMap v1.0 (≈100 M predictions) is hosted on https://huggingface.co/datasets/PatchMap/PatchMap_v1 (accessed on 15 May 2025) HuggingFace under a CC-BY-4.0 license, bundled with loaders, plotting utilities, and a leaderboard script. A 6.5 B-entry v2.0, computed on a dedicated GPU cluster, will be released under the same terms. The code implementation of the project is available on https://github.com/PathMap7/PatchMap GitHub. (accessed on 15 May 2025, used branch master, commit 02bf1b9).

3.8. Why PatchMap?

Its exhaustive spatial coverage (covering a dense grid of all possible locations for path appliance, as explained in Section 3.2) allows researchers to map vulnerability hot-spots, train location-aware detectors, and benchmark placement strategies under identical conditions, and seek to obtain near-optimal results, instead of comparing to other methods. A comparison of PatchMap and other methods is provided in Table 2.

Table 2. Comparison of public adversarial-patch benchmarks. ✓ yes, × no.

Benchmark	Task	l#Imgs	#Patches	Sizes	Loc/img	#Evals	ExhLoc	Cached
ImageNet-Patch [6]	Cls	50 k	10	1 ($50\times 50$ px)	1 †	50 k	×	×
REAP [11]	Det	8433	–	3 ‡	–	14,651	×	×
PatchMap (v1, ours)	Cls	2 k	2 (IDs 2,6)	3 (10/25/50 px)	12,544	150.5 M ⋆	✓	✓

^† One placement per sample with random affine (translation/rotation); no dense per-location map. ^‡ Small/medium/large physical patch sizes (e.g., 10″ × 10″, 10″ × 20″, two 10″ × 20″), rendered with per-sign geometric + lighting transforms. ^⋆ $2000\times 2\times 3\times (112\times 112)=\mathrm{150,528,000}$ placements (stride-2 grid on $224\times 224$). Cached = released precomputed per-location model outputs/maps (rather than only images/transforms). # represents number of.

4. Evaluation Protocol

PatchMap’s dense annotations enable four complementary evaluations.

• Location-wise attack-success heat-maps: For every grid cell $(k,l)$ we compute the clean-correct attack-success rate

  $$\mathrm{ASR}(k,l)=\frac{1}{N_{\mathrm{cc}}}\sum _{i=1}^{N_{\mathrm{cc}}}\mathbb{I}\left[{\widehat{y}}_{i,k,l}\ne y_i\right],$$

  where $N_{\mathrm{cc}}$ is the number of validation images that the model classifies correctly without a patch. The resulting $112\times 112$ heat-map exposes systematic hot- and cold-spots.
• Confidence and calibration shift: Besides logits, PatchMap records soft-max scores, enabling reliability analysis. We report (i) the average confidence drop $\Delta \mathrm{conf}$ as detailed in (3). And (ii) the change in Expected Calibration Error (ECE) and Brier score between clean and patched images.
• Size/conspicuity trade-off: Plotting ASR against patch area across the three sizes yields a Pareto curve that answers: How small can a patch be before its success drops below a chosen threshold?
• Cross-model transfer: Given placements evaluated on model A, we re-score the exact locations on model B and assemble a transfer matrix $T_{A\to B}$. Off-diagonal strength indicates universal spatial vulnerabilities; weak transfer suggests architecture-specific quirks.

• Metrics are averaged over the full validation set and accompanied by $95\%$ bootstrap confidence intervals (1000 resamples). The public code reproduces every figure and table in under 2 GPU-hours on a single V100.
• PatchMap therefore enables fine-grained, statistically sound evaluation of both attacks and defences, opening the door to location-aware robustness research.

5. Analysis and Findings

5.1. Attack-Success Rate (ASR)

For every image $x_i$ that the clean model classifies correctly, and for every patch location $(k,\mathcal{l})$, let ${\tilde{y}}_{i,k,\mathcal{l}}$ denote the predicted label on the patched image $x_i\oplus M_{k,\mathcal{l}}$ and ${\widehat{y}}_i$ the clean prediction. The location-wise ASR is then

$$\mathrm{ASR}(k,\mathcal{l})={\displaystyle \frac{1}{N_{\mathrm{cc}}}}\sum _{i=1}^{N_{\mathrm{cc}}}\mathbb{I}\left[{\tilde{y}}_{i,k,\mathcal{l}}\ne {\widehat{y}}_i\right],$$

where $N_{\mathrm{cc}}$ is the number of clean-correct images.

Optimal location. For each image we also record the highest ASR over all positions; Table 3 reports the mean of this value across the dataset.

Table 3. Mean optimal ASR: fraction of images for which some location causes misclassification.

Patch Size	Patch 2 (“Plate”)	Patch 6 (“Guitar”)
$50\times 50$	0.84	0.94
$25\times 25$	0.79	0.82
$10\times 10$	0.69	0.71

ASR_q. Let $w,h$ be the grid dimensions ($112\times 112$). ASR in quantile q measures the proportion of images fooled at at least a fraction q of all possible locations,

$${\mathrm{ASR}}_q={\displaystyle \frac{1}{N_{\mathrm{cc}}}}\sum _{i=1}^{N_{\mathrm{cc}}}\mathbb{I}\left(\frac{1}{wh}\sum _{k=1}^w\sum _{\mathcal{l}=1}^h[{\tilde{y}}_{i,k,\mathcal{l}}\ne {\hat{y}}_i]>q\right).$$

(2)

Figure 3 plots ${\mathrm{ASR}}_q$ for two representative patches.

Figure 3. ${\mathrm{ASR}}_q$ (2): fraction of images misclassified at more than a q-portion of locations. $AS{R}_q$ indicates the robustness of the effect of the patch on the model with respect to the location of the patch in the image. Patch “Plate” of size $50\times 50$ is the most robust in that sense, as effect is least effected by location obtaining the best results for the worst $\approx 60\%$ of patch locations, “Guitar” of size $50\times 50$ obtains the best results for ≈40% of the best patch locations.

5.2. Confidence Effect

Let $p_{\theta }(y_i\mid x_i)$ be the clean soft-max confidence of the ground-truth class $y_i$. The confidence drop for image i is the maximal change in the model achieved after the attack on any possible location:

$$\Delta {\mathrm{conf}}_i=\underset{k,\mathcal{l}}{max}\left[p_{\theta }(y_i\mid x_i)-p_{\theta }\left(y_i\mid x_i\oplus M_{k,\mathcal{l}}\right)\right],$$

(3)

i.e., the largest fall among all locations. Table 4 shows the mean $\Delta \mathrm{conf}$ per patch size.

Table 4. Average confidence drop $\left(\Delta \mathrm{conf}\right)$ at the worst location per image.

Patch Size	Patch 2	Patch 6
$50\times 50$	0.62	0.71
$25\times 25$	0.60	0.62
$10\times 10$	0.45	0.48

Figure 4 plots the distribution of post-attack confidences; larger patches incur heavier tails towards low confidence, with the “Guitar” patch producing the sharpest degradation at $50\times 50$.

Figure 4. Histogram of soft-max confidence on the ground-truth class after patching.

6. Segmentation-Guided Patch Placement

6.1. Motivation

Most patch attacks search location by gradient descent or reinforcement learning-effective but slow and model-dependent. We observe that semantic-segmentation networks already highlight the pixels most critical for recognizing objects. If a patch occludes those pixels, it should hurt the classifier without any extra optimization. We therefore use segmentation confidence as a fast, architecture-agnostic cue for placing the patch, that proven to be valid when trained on sufficient amount of data [17], despite their limitations [18].

6.2. Approach

Let $g\left(x\right)\in {[0,1]}^{H\times W\times C}$ be per-pixel soft-max scores from a segmentation model, where channel b is background. Define the object-confidence map

$$S=1-g{\left(x\right)}_b.$$

Given a binary patch mask $M\in {\{0,1\}}^{s\times s}$, we slide M over S and pick the location that maximises the summed confidence it covers:

$$(k^{\star },{\mathcal{l}}^{\star })=arg\underset{k,\mathcal{l}}{max}{\left(S\odot {\mathrm{Shift}}_{k,\mathcal{l}}\left(M\right)\right)}_1,$$

(4)

where ⊙ is the element-wise product, ${\mathrm{Shift}}_{k,\mathcal{l}}$ centres the mask at $(k,\mathcal{l})$, and ${(·)}_1$ sums all entries. The chosen centre $(k^{\star },{\mathcal{l}}^{\star })$ is used for every tested classifier; no gradients or queries to the attacked model are required.

The method is demonstrated in Figure 5.

Figure 5. Pipeline of the segmentation-guided placement. A segmentation heat-map (middle) guides the patch (right) to the most confident object region. Colors: blue-green spectrum from low-high values.

6.3. Experimental Setup

We evaluate all placement strategies on the ImageNet-1K validation split, resizing each image to $224\times 224$ pixels. Five classifiers are attacked: ResNet-18 [16], ResNet-50, MobileNet-V2 [19], EfficientNet-B1 [20], Zero-Shot CLIP [21], and a Fast Adversarially Trained ResNet-50 [22]. Patch placement is guided by DeepLab-v3+ [23] with a ResNet-101 backbone, pretrained on PASCAL-VOC 2012, whose segmentation confidences serve as a zero-gradient cue. All experiments can be conducted with a single processor with at least 8 GB of memory (with/without GPU).

Unless stated otherwise, the adversary is the “Plate” universal patch from ImageNet-Patch [6], resized to $50\times 50$ (≈4.98% of the image) and without rotation. We benchmark three optimisation-free placement rules: (i) Random-uniformly sampling the patch centre, (ii) Fixed-choosing the best of four preset offsets from the image centre, and (iii) our Seg-guided heuristic described in (4).

Performance is reported as attack-success rate (ASR) on the subset of images that each model classifies correctly in the clean setting.

6.4. Overall Performance

Table 5 compares attack-success rates (ASR) for three optimisation-free placement strategies. Across all four ImageNet-trained architectures, the segmentation-guided (Seg-guided) heuristic achieves the highest ASR, improving on Random placement by an average of 8 pp and outperforming the Fixed four-offset baseline by 13 pp. Crucially, the advantage persists on the adversarially trained ResNet-50, where Seg-guided placement still raises ASR from 0.31–0.36 to 0.39, underscoring that robust training alone is not sufficient to counter strategic patch positioning. Table 6 compares the average confidence drop $\Delta \mathrm{conf}$ for one patch of sizes $25\times 25,50\times 50$ and one model.

Table 5. ASR of optimisation-free placement strategies. Patch resized with no rotation. Bold: the optimal result for each setup.

Model	Random	Fixed	Seg-Guided
“Plate”	$50\times 50$		$(\approx$4.98%)
Improvement			0.052
ResNet-50	0.39	0.32	0.46
+ Adverserial Training [22]	0.36	0.31	0.39
ResNet-18	0.57	0.46	0.63
MobileNet-V2	0.48	0.41	0.55
EfficientNet-B1	0.38	0.34	0.42
CLIP	0.44	0.43	0.47
“Plate”	$25\times 25$		$(\approx$1.25%)
Average Improvement			0.042
ResNet-50	0.19	0.15	0.21
+ Adverserial Training [22]	0.11	0.09	0.14
ResNet-18	0.22	0.25	0.29
MobileNet-V2	0.35	0.32	0.37
CLIP	0.37	0.35	0.43
“Electirc Guitar”	$50\times 50$		$(\approx$4.98%)
Average Improvement			0.046
ResNet-50	0.32	0.27	0.36
+ Adverserial Training [22]	0.24	0.17	0.3
ResNet-18	0.44	0.37	0.49
MobileNet-V2	0.48	0.42	0.52
CLIP	0.39	0.30	0.43
“Electirc Guitar”	$25\times 25$		$(\approx$1.25%)
Average Improvement			0.030
ResNet-50	0.19	0.16	0.22
+ Adverserial Training [22]	0.09	0.07	0.12
ResNet-18	0.25	0.22	0.28
MobileNet-V2	0.33	0.31	0.36
CLIP	0.37	0.32	0.41
“Typewriter Keyboard”	$50\times 50$		$(\approx$4.98%)
Average Improvement			0.058
ResNet-50	0.33	0.28	0.37
+ Adverserial Training [22]	0.21	0.14	0.26
ResNet-18	0.43	0.37	0.48
MobileNet-V2	0.47	0.42	0.52
CLIP	0.38	0.31	0.41
“Typewrtier Keyboard”	$25\times 25$		$(\approx$1.25%)
Average Improvement			0.024
ResNet-50	0.18	0.15	0.20
+ Adverserial Training [22]	0.10	0.07	0.12
ResNet-18	0.23	0.19	0.25
MobileNet-V2	0.33	0.31	0.35
CLIP	0.35	0.28	0.39

Table 6. $\Delta \mathrm{conf}$ of optimisation-free placement strategies on ResNet-50. Patch resized with no rotation. Bold: the optimal result for each setup.

Patch	Size	Random	Fixed	Seg-Guided
Plate	$50\times 50$	0.63	0.52	0.69
Plate	$25\times 25$	0.38	0.32	0.42

6.5. Effect of Patch Size

Figure 6, Figure 7 and Figure 8 plots ASR against patch side length on a representative ResNet-18 for lengths $\{5,15,25,50\}$. All methods degrade as the patch shrinks, yet Seg-guided placement maintains a consistent margin: with a $10\times 10$ px patch-covering barely 2% of the image-it still fools 27% of clean-correct samples, versus 18% for Random and 14 % for Fixed. This resilience indicates that the segmentation heat-map is a reliable cue even when the patch is almost imperceptible.

Figure 6. ASR on ResNet-18 versus patch size for patch ‘Plate’.

Figure 7. ASR on ResNet-18 versus patch size for patch ‘Typewriter’.

Figure 8. ASR on ResNet-18 versus patch size for patch ‘Electric Guitar’. Optimal placement remains crucial as size shrinks.

6.6. Correlation with Confidence Drop

We examine how segmentation confidence predicts the eventual confidence drop of the classifier at each location. Define overall object confidence

$$p_{\mathrm{seg}}(\mathrm{object}\mid x)=1-g{\left(x\right)}_b$$

(5)

cf. Equation (5). Figure 9 examines the correlation between segmentation confidence (as detailed previously) and the effect on the ground-truth class confidence $\Delta \mathrm{conf}$ for several possible patches. There is a strong correlation for scores above 0.2, especially for smaller patches, confirming that the segmentation map is a useful but not perfect proxy for the worst-case location. A possible explanation for that phenomenon is that lower confidence effect of the patch is obtained due to the addition of a new object to the image (the adversarial patch), and not by changing the object itself. Thus, in certain areas of the image, the object is covered by an adversarial patch which makes no further effect than a patch besides the object.

Figure 9. Correlation between segmentation confidence and classifier confidence drop ($\Delta \mathrm{conf}$).

6.7. Qualative Examples

Figure 10 demonstrates the connection between the segmentation probability prediction by DeepLabV3Plus with a Resnet-101 backbone as detailed in Section 6.3 as not object (i.e., $1-P_{\theta }\left(clas{s}_0\right)$ where $clas{s}_0$ is background), and the empirical map of $\Delta con{f}_y$ i.e., the effect on the prediction probability of the ground-truth class y measured on ResNet-18, after appliance of patch “Plate” of size $50\times 50$. Most locations which yield the highest effect are indeed part of the predicted object, much of the predicted object is does not yield the highest $\Delta \mathrm{conf}$ values.

Figure 10. Left to Right: Original Image, Segmentation confidence prediction, Predicted confidence drop map, each pixel represents the confidence when the adversarial patch is applied at that location.

For large objects, there are distinctive parts that define the object (For example, the head of a dog is important in defining and identifying that it is indeed a dog), these areas show a higher value of $\Delta con{f}_y$ meaning that these areas are more vulnerable to patch placement. Compared to other placement methods such as in [11] prioritizing path placement on the object, they like our method miss the optimal patch placement in a significant portion of the images.

7. Conclusions

We have introduced PatchMap, the first large-scale, spatially exhaustive benchmark for adversarial patch placement. By evaluating over $1.0\times {10}^8$ patch placements on ImageNet–1K validation images, PatchMap uncovers consistent “hot-spots” of vulnerability and quantifies how attack success and confidence collapse vary across both location and patch size.

Building on this dataset, we proposed a simple, segmentation-guided placement heuristic that leverages off-the-shelf semantic masks to select high-impact regions without any gradient queries or fine-tuning of the target model. Across five architectures-including adversarially trained ResNet-50-our method yields an 8–13 pp improvement in attack-success rate over random or fixed baselines, demonstrating that zero-gradient spatial cues can substantially amplify patch efficacy.

Looking forward, PatchMap opens the door to a new class of location-aware defenses and attacks, allowing for methods to find the optimal patch location with respect to universal patch attack, in new domains such as medical [24] or robotics [25]. Other direction is to understand the relation between input patch attacks and bit-flip attacks [26], as those are the most practical in real world setup.

Future releases (v2.0) will scale to 6.5 billion placements and incorporate diverse backbones (ViT, ConvNeXt and modified architectures [27]), enabling cross-family transfer studies. We anticipate that researchers will build on PatchMap to design spatially informed detectors, robust training curricula, and context-dependent attacks across classification, detection, and beyond, advancing the security and reliability of vision systems.

7.1. Task Scope

Our benchmark focuses on ImageNet-style image classification. Extending PatchMap to detection and segmentation is non-trivial, since patch placement interacts with multi-instance scenes, localization objectives, and different evaluation metrics (mAP/IoU). We view this as an important direction for future releases.

7.2. Segmentation Prior

Our segmentation guided placement uses a single pretrained segmentation model (DeepLab-v3) trained on PASCAL-VOC. While this choice keeps the method simple and reproducible, the strength of the placement signal may vary across segmentation backbones, training datasets, and newer foundation segmentation models.

7.3. Patch Transforms and Physical Realism

PatchMap v1 uses static, axis aligned square patches without rotation. This setting is sufficient to reveal systematic spatial hot-spots, but it does not capture physical effects such as rotation, perspective deformation or illumination changes. Incorporating these transforms would enable more realistic benchmarking of physical-world attacks.