Boosting the Transferability of Adversarial Attacks with Semantic-Invariance and Low-Gradient Replacement

Abstract

Deepneural networks are highly vulnerable to adversarial examples, which are generated by introducing subtle perturbations to input data to mislead model classification. Currently, transfer-based attacks are prevalent in adversarial example generation and can be categorized into input transformation-based and gradient-based methods. However, most input transformation-based methods tend to produce augmented replicas that are semantically inconsistent with the original inputs, while gradient-based methods often leave low-gradient regions unperturbed within the model’s critical attention areas. These limitations constrain further improvements in adversarial transferability. In this work, we propose a Semantic-Invariance and Low-Gradient Replacement Method (SLRM) to address these challenges. Our framework integrates semantically consistent augmentation and gradient replacement as follows: (1) a feature extractor captures semantic features from original inputs and a reconstructor generates augmented replicas that preserve semantic fidelity to enhance input diversity, and (2) low-gradient regions in adversarial examples are systematically replaced with corresponding regions from augmented replicas to eliminate the under-perturbed areas critical for model robustness. Comprehensive empirical evaluations on ImageNet demonstrate that SLRM significantly enhances the transferability of baseline methods and seamlessly integrates with state-of-the-art approaches to further improve their performance. Moreover, SLRM substantially improves the robustness of baseline methods against defended models, achieving superior attack success rates under advanced adversarial defenses.

1. Introduction

Deep Neural Networks (DNNs) have demonstrated remarkable efficacy across a diverse spectrum of application domains, particularly in the field of computer vision [1,2]. However, DNNs are known to be vulnerable to adversarial examples, i.e., nearly imperceptible perturbations to input data that can induce classification errors [3,4,5,6,7,8]. The prevalence of adversarial examples in security-critical applications poses significant risks to model reliability. Consequently, the study of adversarial attack mechanisms can reveal vulnerabilities in DNN models and facilitate robust defense strategies.

The existing literature typically categorizes adversarial attacks into two paradigms: white-box and black-box attacks. White-box attacks generate adversarial examples with full knowledge of the model, including its architecture and parameters [4,9]. However, obtaining such comprehensive information about the target model is almost impossible in real-world applications. Thus, black-box attacks, which do not require prior knowledge of the underlying architecture and parameters, pose a greater threat in practical applications [10,11].

The most popular black-box attacks are transfer-based attacks [12,13], which generate adversarial examples using surrogate models. These attacks are broadly categorized into input transformation-based and gradient-based approaches. Input transformation-based attacks enhance the transferability of adversarial examples by transforming the original inputs, e.g., random resizing [14] and translation [15]. However, these transformations lack sufficient diversity. To address this, several variants have been proposed. For example, SIM [16] leverages multiple scaled replicas of the original input for enhancement, while Admix [17] mixes replicas from other image classes with the original input. Image super-resolution [18], which recovers high-resolution images from low-resolution inputs, is another technique that may be used in this context. Nevertheless, the replicas generated by these methods may be semantically inconsistent with the original input [19]. Gradient-based attacks, such as FGSM [4] and I-FGSM [9], craft adversarial examples by applying perturbations along the gradient direction. However, they usually overfit the surrogate models, resulting in poor transferability. To enhance transferability, researchers have introduced various approaches to refine gradient computation (e.g., MIM [10] and NIM [16]). As these methods iteratively accumulate perturbations along the gradient direction, they modify the input’s gradient distribution, resulting in elevated gradient magnitudes. However, we observe that some low-gradient areas persist within the attention areas of generated adversarial examples. These low-gradient areas in the adversarial examples may correspond to under-perturbed regions, a phenomenon also observed in [20].

To address the inherent limitations of existing input transformation-based and gradient-based attacks, we propose a novel transfer-based adversarial attack framework, termed the Semantic-Invariance and Low-Gradient Replacement Method (SLRM). In the semantic-invariance module, an extractor is employed to extract semantic features from the original input, and a reconstructor is trained to reconstruct a semantically invariant replica from these features for input transformation. Additionally, we propose gradient replacement to boost the low gradients of adversarial examples in the model’s attention areas, thereby enhancing their transferability. Specifically, we first identify the low-gradient areas in the replica and then replace their gradient values with those from the corresponding locations in the adversarial example. Experimental results demonstrate that our approach enhances the transferability of state-of-the-art attacks against both standard (undefended) and defended black-box models. The key contributions of our method are summarized as follows:

• We propose a novel perspective for interpreting the transferability of attacks: low-gradient areas in adversarial examples likely correspond to under-perturbed regions, which represent the model’s vulnerable regions. Accordingly, we employ a gradient replacement strategy to amplify the low gradients of the adversarial examples in the model’s attention areas. As shown in Section 4.9, for both benign samples and MIM adversarial examples, significant gradients are concentrated on the main subject area, whereas gradients in the background (e.g., the black regions around the panda) and non-critical texture regions (e.g., the fur near its facial features) are close to zero, as shown in blue in the first and second rows. Our method appears to enhance the model’s sensitivity to these low-gradient regions, as indicated by the increased gradient magnitudes in the heatmaps.
• We explore the connection between semantic consistency and the transferability of adversarial examples. Our experimental results indicate that preserving semantic consistency can further improve transferability.
• We conduct comprehensive experiments showing that SLRM significantly improves the transferability across various black-box models including security models. Furthermore, SLRM can be seamlessly integrated with existing state-of-the-art (SOTA) attacks to further improve their performance.

2. Related Work

This section presents a concise overview of the current methods for both adversarial attacks and corresponding defense mechanisms.

2.1. Adversarial Attacks

Various white-box attack methods [3,4,9,21,22] have been proposed to craft adversarial examples that cause misclassification in DNNs. The performance of these attacks is severely limited when facing black-box models. Therefore, attacks against black-box models have been proposed, which are mainly transfer-based attacks [5,23,24,25,26,27,28] that use surrogate models to generate adversarial examples. In this subsection, we introduce two branches of transfer-based attacks.

Input transformation-based attacks. The effectiveness of data augmentation has been widely recognized in enhancing model generalization, and has also been exploited to improve adversarial transferability. Ref. [14] adjusted the size of input samples through random scaling or padding to obtain enriched input information. Ref. [15] proposed the translation-invariant attack, which translates input images to achieve gradient translations and averages multiple operations to strengthen the transferability of adversarial examples. However, these simple transformations of the original inputs lacked sufficient diversity. Subsequently, several advanced transformation methods were proposed, generating multiple replicas of the original inputs for transformation. Ref. [16] proposed the scale-invariant attack, which generates a set of replica images by scaling the original input and computes the average gradient of this set to optimize the update of adversarial examples. Ref. [17] suggested creating a set of replica images by scaling the original input and mixing it with random images from other classes. Ref. [11] introduced the use of spectral transformations to improve the transferability of adversarial examples. Ref. [13] proposed incorporating style patterns from other domains into the original input samples to create diversified replicas, enhancing the transferability. Ref. [29] proposed segmenting the input image into blocks and randomly swapping and rotating them to improve the transferability of adversarial examples. The recent SMP-Attack [30] partitions the input image into a set of multi-granularity patches, thereby enhancing the quality of the computed aggregate gradient. Unfortunately, these advanced transformation-based attacks tend to generate replicas that are semantically inconsistent with the original input, which may limit the transferability of adversarial examples.

Gradient-based attacks. The FGSM [4] crafts adversarial examples by adding perturbations in the direction of the gradient update. I-FGSM [9] constructs adversarial examples iteratively by adding small perturbations at each iteration. Unlike I-FGSM, ref. [10] incorporated momentum into each iteration, stabilizing the update direction by accumulating gradients from previous iterations, thereby avoiding local optima and generating adversarial examples with stronger transferability. Ref. [16] introduced Nesterov gradient acceleration into the gradient computation of I-FGSM. Compared to momentum, it not only stabilizes the update direction but also exhibits a forward-looking property, resulting in better transferability. Ref. [31] optimized the gradient update of adversarial examples by sampling multiple points along the previous gradient direction, averaging these sampled gradients, and accumulating them into the current gradient. Ref. [32] leveraged the skip connections of residual networks to incorporate lower-level gradient information during gradient computation, reducing the influence of residual block gradients and improving the transferability of adversarial examples. Ref. [12] observed that existing gradient optimization algorithms tend to halt at certain peaks, which often overfit the surrogate model. To address this, they introduced a penalized loss function into the original loss function of adversarial algorithms, aiming for adversarial examples to move toward flatter regions, thereby achieving better transferability. Ref. [33] proposed a global momentum initialization method to suppress gradient elimination. Ref. [34] proposed an Adaptive Feature Fusion Module (AFFM) that mitigates the contradictions between CNNs and Transformers in terms of dimensions and representativeness, leading to better perturbation generation and enhanced general attack capability. Ref. [35] proposed a novel Staircase Sign Method that divides the gradient sign into several segments based on gradient unit values and assigns a staircase weight to each segment to craft more effective adversarial perturbations. However, these advanced gradient-based attacks still have some low gradients in the attention areas of their adversarial examples, which may affect the adversarial transferability.

2.2. Adversarial Defense

A range of adversarial defense strategies have been developed to mitigate the impact of adversarial examples on neural network models. Among all defense strategies, adversarial training is undoubtedly the simplest and most direct approach. It improves model robustness by adding specific adversarial examples to the training dataset and retraining the model. Ref. [4] first proposed the adversarial training paradigm, in which models are trained using a mixture of clean samples and adversarial examples generated by FGSM. However, models trained using this approach lack generalization and are only robust against adversarial examples crafted by FGSM. Ref. [36] observed that adversarial examples generated by the same attack algorithm differ across models. By utilizing all these adversarial examples in adversarial training, richer information can be obtained compared to traditional methods. Consequently, they proposed the concept of ensemble adversarial training, which involves generating adversarial examples from multiple models and using them for training. This approach significantly enhances model robustness and improves performance against black-box transfer attacks. Ref. [37] introduced the idea of training an advanced denoiser to cleanse adversarial perturbations from input images (HGD). Ref. [38] suggested employing an end-to-end image compression model to mitigate adversarial perturbations (Comdefend). Ref. [39] trained an image purification model based on self-supervised neural networks, optimizing its performance by iteratively inputting adversarial and original images (NRP). Ref. [40] proposed adding Gaussian random noise to the training dataset to enhance the model’s capability to recognize noisy images, with final predictions determined based on multiple noise test results (RS). Ref. [41] utilized a diffusion model to add noise during the forward diffusion of adversarial examples, and then reconstructed clean inputs through reverse generation (Diffpure). In this study, we employ these defense models to evaluate the efficacy of our proposed approach.

3. Method

In this section, we first present an architecture diagram (Figure 1) that summarizes the overall methodology of our proposed SLRM method. We then elaborate on the method details in the subsequent subsections.

Figure 1. Model architecture of SLRM.

3.1. Motivation

Numerous studies have demonstrated that data augmentation can enhance the transferability of adversarial examples. Currently, most advanced input transformation-based attack methods enhance adversarial transferability by generating augmented replica images through data augmentation to obtain richer gradient information. For instance, SIM [16] generates augmented replicas by applying multiple scaling operations to the original input image; Admix [17] creates augmented replicas by superimposing images from other categories onto the original input; and STM [13] trains a style transformer to fuse style replicas from different domains into the original input, thereby improving adversarial transferability. However, we observe that these attacks occasionally incorporate replicas semantically inconsistent with the original input. Such inconsistency may degrade the accuracy of gradient computation, thus limiting further improvements in adversarial transferability.

Gradient-based attack methods primarily add perturbations to regions with significant gradient variations, leading to adversarial examples where perturbations concentrate in high-gradient areas. In our empirical research, we find persistent low-gradient regions in these adversarial examples, which likely correspond to minimal perturbation areas. Notably, some of these low-gradient regions overlap with model attention areas that are critical for correct classification, which helps explain why the attacks remain unsuccessful. Therefore, unperturbed low-gradient regions may constrain further improvements in adversarial transferability.

To overcome these limitations in transfer attacks, we propose a novel adversarial attack method: the Semantic-Invariance and Low-Gradient Replacement Method (SLRM). First, we ensure semantic consistency between augmented replicas and original inputs during replica generation to preserve gradient computation accuracy. Second, we perform low-gradient region replacement in adversarial examples, effectively applying input transformations to these regions to suppress the presence of low-gradient areas. The integration of these two operations effectively eliminates the low-gradient regions in adversarial examples, thereby further enhancing their transferability.

3.2. Preliminaries

Notation. Denote the ground-truth label of the benign input image x by y, and define $f(x;\theta )$ as a classifier that maps x to y. Let $J(·,·)$ be the loss function associated with $f(x;\theta )$. Denote the adversarial example of input x by $x^{adv}$. $x^{adv}$ satisfies ${\parallel x^{adv}-x\parallel }_{\infty }\le ϵ$, where $ϵ$ is the perturbation budget.

Gradient-based attacks improve transferability by introducing momentum into the gradient computation. For example, the update rule for the adversarial example in MIM [10] is given by

$$g_{t+1}=\mu ·g_t+{\displaystyle \frac{{\nabla }_{x_t^{adv}}J(f(x_t^{adv};\theta ),y)}{{\parallel {\nabla }_{x_t^{adv}}J(f(x_t^{adv};\theta ),y)\parallel }_1}},$$

(1)

and

$$x_{t+1}^{adv}=Cli{p}_x^{ϵ}\left\{x_t^{adv}+\alpha ·sign\left(g_{t+1}\right)\right\},$$

(2)

where $g_t$ is the gradient at iteration t, $\mu$ is the decay factor of $g_t$, $\alpha$ is the step size of the iteration, and the $Clip(·)$ function constrains the perturbation of $x_t^{adv}$ within the $ϵ$-ball.

3.3. Semantic Invariance

In our work, we use a semantic extractor $E(·)$ to extract semantic features, and a feature reconstructor $R(·)$ to reconstruct a semantically consistent replica $x^{\prime }=R\left(E\left(x\right)\right)$ from them. Inspired by [42]’s research, we utilize convolutional neural networks to extract the semantic features of images.

Specifically, we adopt the shallow convolutional layers of a pretrained model (e.g., vgg19-bn [43]) as the feature extractor $E(·)$, which is used to extract semantic features $s=E\left(x\right)$ from the original input x. Subsequently, these features are processed by a feature reconstructor $R(·)$, designed as a mirror network of the extractor, to generate a reconstructed replica. Within $R(·)$, transposed convolution and upsampling operations are adopted to reduce the channel dimension of the feature maps. Activation functions are applied to all transposed convolutional layers, except the final one, ensuring that the output pixel values remain within the range $[0,1]$. The training objective of $R(·)$ is to produce a replica that preserves semantic consistency with the original input. Its loss function is formulated as follows:

$$L={\parallel E\left(R\right(s\left)\right)-s\parallel }_2.$$

(3)

We evaluate the semantic consistency between SLRM-generated replicas and original images across five models using the ImageNet dataset [44]. In Table 1, the first row presents the attack success rate (ASR) of original images on the five models, the second row displays the ASR of images perturbed by Gaussian random noise (intensity: $16/255$), and the third row shows the ASR of SLRM-generated replicas. Comparing the first and third rows, the SLRM-generated replicas demonstrate superior preservation of semantic consistency with the original images compared to their Gaussian-noise-perturbed counterparts.

Table 1. Attack success rate (%) of replicas when different strategies are used.

Images	Inc-v3	Inc-v3adv	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
Clean	4.9	4.9	6.1	8.2	2.2
Noise	23.6	44.0	28.7	34.6	23.8
SLRM	7.7	7.8	8.8	11.3	4.2

In addition, we further assess the image similarity between attack-generated replicas and original inputs. Table 2 reports the average SSIM [45], PSNR, and LPIPS [46] between original inputs and augmented replicas (averaged over 10 iterations) for SIM, Admix, STM, and our SLRM. SSIM and PSNR are widely adopted similarity metrics, where higher scores indicate lower distortion and better similarity, while lower LPIPS values correspond to higher perceptual similarity. As demonstrated in this table, the SLRM-enhanced replicas achieve significantly higher similarity to the original inputs than the replicas from SIM, Admix, and STM.

Table 2. Semantic similarity between the original images and the replicas generated by four different methods.

Images	SSIM ↑	PSNR ↑	LPIPS ↓
SIM	0.5934	13.61	0.3925
Admix	0.5567	12.79	0.4493
STM	0.5472	14.83	0.4332
SLRM (Ours)	0.8782	31.83	0.1362

↑: higher is better; ↓: lower is better.

3.4. Gradient Replacement

In TIM [15], the authors found that the defense model has different attention areas compared to the normal model. Therefore, they proposed to modify the gradient of the adversarial example via gradient translation, thereby improving its transferability. In our work, we replace the gradients in low-gradient areas of the replica $x^{\prime }$ with those from the corresponding areas of the adversarial example $x^{adv}$ in order to suppress the presence of low gradients in the model’s attention areas and achieve gradient transformation. To clarify, this process can be interpreted as adjusting the direction of gradients and amplifying the low gradients of the adversarial example.

First, we compute the q-th sample quantile (i.e., the replacement ratio) of the absolute values of gradients g over the flattened full tensor. The cut-off value $\xi$ is given by

$$\xi =F^{-1}\left(q\right)=inf\left\{\right|g|:F(\left|g\right|)\ge q\},$$

(4)

where $F(·)$ is the empirical cumulative distribution function of the absolute gradient values computed over the flattened full tensor.

Subsequently, we record the positions in the replica where the absolute gradient value is less than $\xi$ using a mask matrix $M^{\left(c\right)}={\left(m_{h,w}^{\left(c\right)}\right)}_{H\times W}\in {\{0,1\}}^{H\times W}$, which is defined element-wise as follows:

$$m_{h,w}^{\left(c\right)}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}|g_{h,w}^{\left(c\right)}|\le \xi \hfill \\ 0\hfill & \mathrm{if}|g_{h,w}^{\left(c\right)}|>\xi \hfill \end{array}\right.,c=1,2,3;h=1,\dots ,H;w=1,\dots ,W$$

(5)

where c is the channel index, and H and W are the height and width of each channel. Then, the complementary matrix of the mask matrix $M^{\left(c\right)}$ is

$$C^{\left(c\right)}\triangleq {\mathbf{1}}_H{\mathbf{1}}_W^{\mathrm{T}}-M^{\left(c\right)},c=1,2,3,$$

(6)

which marks the entries with significant gradients.

The details of our proposed SLRM can be found in Algorithm 1. Firstly, we use the extractor $E(·)$ to capture the semantic features of the original input x. Secondly, we use the reconstructor $R(·)$ to reconstruct these features into a semantically invariant replica $x^{scc}$. Then, we calculate the gradients of the adversarial example $x^{adv}$ and the replica $x^{scc}$, and mix them with a ratio of $\delta$. Next, we identify the low-gradient areas in the replica using Equations (5) and (6) and replace them with the gradients of the adversarial example. Finally, we update the adversarial example and the replica using Equation (1). In our algorithm, the semantic-invariance module provides a data augmentation replica that differs slightly from the original input x but keeps semantic consistency with it. The clip operator restricts both $x^{adv}$ and $x^{scc}$ to be within the $ϵ$-ball of x. This restriction ensures that during the update, both the adversarial example $x^{adv}$ and the replica $x^{scc}$ remain semantically consistent, and $x^{scc}$ provides diverse information to $x^{adv}$. The gradient mixing and replacement amplify the low gradients of $x^{adv}$ in the model’s attention areas and further ensure that $x^{scc}$ does not deviate too far from $x^{adv}$.

Algorithm 1 Semantic-Invariance and Low-Gradient Replacement Method (SLRM)

Input:  A benign image x, its ground-truth label y, and replica $x^{\prime }$; surrogate classifier $f(·;\theta )$ with parameters $\theta$, loss function $J(·,·)$; replacement ratio q, mixing coefficient $\delta$, perturbation budget $ϵ$, number of iterations T, decay factor $\mu$. Output:  An adversarial example $x^{adv}$.   1: Initialize $x_0^{adv}=x$; $x_0^{scc}=x^{\prime }$; $g_0=0$; $g_0^{scc}=0$; $\alpha =ϵ/T$;   2: for $t=0$ to $T-1$ do   3:    Compute the gradients of $x_t^{adv}$ and $x_t^{scc}$: $$\begin{array}{c}\hfill {\overline{g}}_{t+1}={\nabla }_{x_t^{adv}}J(f(x_t^{adv};\theta ),y),{\overline{g}}_{t+1}^{scc}={\nabla }_{x_t^{scc}}J(f(x_t^{scc};\theta ),y);\end{array}$$   4:    Mix the gradients with coefficient $\delta$: $$\begin{array}{c}\hfill {\tilde{g}}_{t+1}=\delta ·{\overline{g}}_{t+1}+(1-\delta )·{\overline{g}}_{t+1}^{scc};\end{array}$$   5:    Get the mask matrix of ${\overline{g}}_{t+1}^{scc}$ and its complement using Equations (5) and (6);   6:    Compute ${\tilde{g}}_{t+1}^{scc}$ by replacing low-gradient regions of ${\overline{g}}_{t+1}^{scc}$ with ${\tilde{g}}_{t+1}$: $$\begin{array}{c}\hfill {\tilde{g}}_{t+1}^{scc}=M_{t+1}\odot {\tilde{g}}_{t+1}+C_{t+1}\odot {\overline{g}}_{t+1}^{scc};\end{array}$$   7:    Update momentum-based gradients: $$\begin{array}{c}\hfill g_{t+1}=\mu ·g_t+{\displaystyle \frac{{\tilde{g}}_{t+1}}{{\parallel {\tilde{g}}_{t+1}\parallel }_1}},g_{t+1}^{scc}=\mu ·g_t^{scc}+{\displaystyle \frac{{\tilde{g}}_{t+1}^{scc}}{{\parallel {\tilde{g}}_{t+1}^{scc}\parallel }_1}};\end{array}$$   8:    Update examples via sign gradient ascent and ${\mathsf{\ell }}_{\infty }$ clipping: $$\begin{array}{c}\hfill x_{t+1}^{adv}=Cli{p}_x^{ϵ}\left\{x_t^{adv}+\alpha ·sign\left(g_{t+1}\right)\right\},x_{t+1}^{scc}=Cli{p}_x^{ϵ}\left\{x_t^{scc}+\alpha ·sign\left(g_{t+1}^{scc}\right)\right\};\end{array}$$   9: end for 10: return The final adversarial example $x^{adv}=x_T^{adv}$.

3.5. Efficiency

Our proposed SLRM contains two modules: the semantic-invariance module and the low-gradient replacement module. Therefore, its efficiency depends on these two approaches. Regarding the semantic-invariance module, the encoder $E(·)$ is initialized with the pretrained vgg19-bn and its weights are frozen during training. Only the parameters of the reconstructor $R(·)$ are updated. It can be implemented as a “mirror network” of the vgg19-bn shallow layers and involves a one-time offline training cost. Once trained, its weights are frozen and reused across all attacks, making this cost negligible for large-scale robustness evaluation. For the low-gradient replacement module, SLRM performs one forward/backward pass for the adversarial example and one for its semantic replica at each iteration, as well as quantile thresholding and mask generation. These computations are highly parallelizable and incur minimal GPU overhead. As shown in Table 3, integrating SLRM increases the per-image generation time by only 8.93–$12.90\%$ across eight representative attack methods and peak GPU memory by 38–$47\%$, primarily due to storing the additional replica. The absolute runtime overhead scales proportionally with the complexity of the base attack, confirming that the overall computational burden is largely determined by the underlying baseline.

Table 3. Time and peak memory overhead per image for various attack methods.

Attack	Time (s/img)		Overhead (%)	Peak Mem (MB)		Mem Overhead (%)
	w/o SLRM	w/SLRM		w/o SLRM	w/SLRM
DIM	0.31	0.35	12.90	1482	2063	39.20
TIM	0.32	0.35	9.38	1632	2355	44.30
SIM	0.36	0.40	11.11	1713	2477	44.60
Admix	1.68	1.83	8.93	2147	2981	38.84
TDM	0.34	0.38	11.76	1693	2416	42.71
PGN	1.13	1.26	11.50	5846	8069	38.03
STM	5.30	5.98	12.83	9001	12,984	44.25
BSR	1.52	1.67	9.87	7483	11,028	47.37

3.6. Theoretical Analysis

From an optimization perspective, crafting the adversarial example $x^{adv}$ aims to maximize the loss $J(f,y)$ of the surrogate model f, thereby deceiving the classifier. Due to the high nonlinearity of DNNs, the surrogate model’s loss surface contains many local extrema, some of which may not correspond to truly transferable adversarial regions, leading to failed attacks. To improve attack transferability and reduce overfitting to the surrogate model, it is beneficial to break the surrogate model’s reliance on specific gradient directions during gradient-based attacks. To this end, we propose a low-gradient replacement strategy. The theoretical justification for this strategy is presented in the following subsection.

Proposition 1. 

Under Assumptions A1–A3 (given in Appendix A), given a perturbation budget $ϵ>0$, let $x^{adv}$ denote the adversarial example generated by Algorithm 1, and let ${\tilde{x}}^{adv}$ be the adversarial example generated by the attack method without the gradient replacement operation. Then,

$$\mathbb{E}\left[J(f(x^{adv};\theta ),y)\right]\le \mathbb{E}\left[J(f({\tilde{x}}^{adv};\theta ),y)\right]-\Delta$$

where $\Delta >0$ is a constant.

Remark 1. 

This proposition formalizes the trade-off between attack efficacy and transferability. Low-gradient replacement sacrifices attack performance on the surrogate model, with Δ quantifying the strict decrease in expected loss. This sacrifice of optimality is precisely what prevents overfitting to the surrogate model, trading surrogate-specific optimality for better black-box generalization. The magnitude of Δ depends on the flat region ${\mathcal{H}}_{\delta }$ and the local geometry of the surrogate loss, and it is guaranteed to be positive when the perturbation budget ϵ is chosen to be sufficiently small. See Appendix A for the proof of this proposition.

4. Experiment

In this section, we perform extensive experiments on an ImageNet-compatible dataset to validate the effectiveness of the proposed SLRM. We begin by detailing the experimental setup. Next, we compare the proposed method with various baselines on both standard and defensive models. Finally, we conduct ablation studies to analyze the impact of the hyperparameters in our SLRM.

4.1. Experimental Setup

Hardware Used. All experiments were conducted on NVIDIA 4090 GPUs.

Dataset. To ensure a fair comparison, we employ the NIPS2017 competition dataset, which consists of 1000 images randomly selected from the ImageNet dataset [44] and has been widely adopted in adversarial example research [12,13,16,17]. All images are normalized by dividing the original pixel values by 255, scaling their range from $[0,255]$ to $[0,1]$.

Models. To evaluate the effectiveness of our method, we consider diverse structured network architectures, including CNN and Transformer variants. For our CNN-based architectures, we select six standard trained models widely adopted in adversarial example research: ResNet-50 (Res-50) [2], VGG-19 [43], MobileNet-v2 (Mob-v2) [47], Inception-v3 (Inc-v3) [48], ResNeXt101 (ResX101) [49], and ConvNeXt [50]. For our Transformer-based models, we incorporate four state-of-the-art architectures: Vision Transformer-base/16 (ViT-B) [51], DeiT-Small (DeiT-S) [52], EfficientNet-B4 (Efficient) [53], and RVT-Tiny (RVT-T) [54]. Furthermore, we evaluate our method against nine state-of-the-art defense techniques that are robust against black-box attacks on the ImageNet dataset, including four adversarially trained models (i.e., Inc-v3_adv [55], Inc-v3_ens3, Inc-v3_ens4, and IncRes-v2_ens [36]) and five specialized defenses: HGD [37], ComDefend [38], RS [40], NRP [39], and DiffPure [41].

Baseline. We integrate our method into various widely used adversarial attack baselines, including DIM [14], TIM [15], SIM [16], and Admix [17]. We also integrate our method into the SOTA gradient-based attack PGN [12] and the SOTA input transformation-based attacks STM [13] and BSR [29], respectively.

Semantic-Invariance Module. For the semantic-invariance module, we employ the shallow convolutional layers of vgg19-bn [43] (i.e., conv1_1, conv1_2, and conv2_1) as the extractor $E(·)$ with pretrained weights. The reconstructor $R(·)$ is built as a symmetric network of the extractor, consisting of three transposed convolutional layers (kernel size $3\times 3$, stride 1, and padding 1). We use ReLU activation after each transposed convolutional layer, except for the final layer, which uses a sigmoid function to constrain output pixel values to $[0,1]$. Moreover, we randomly sampled 30,000 images from ImageNet [44] to train the reconstructor by minimizing the loss function Equation (3) using the Adam optimizer with the learning rate set to $1\times {10}^{-5}$.

Hyperparameters. In this work, for all attacks, we set the perturbation budget $ϵ=16.0/255$, the number of iterations $T=10$, and the decay factor $\mu =1.0$, which are standard settings adopted in the majority of the adversarial attack literature. For TIM, we set the kernel size $k=7$. For DIM, we set the transformation probability $p=0.5$. For SIM and Admix, we set the number of replicas $m_1=5$, the number of mixed samples $m_2=3$, and the admix ratio $\eta =0.2$. For STM, we set the mixing ratio $\gamma =0.5$, the noise upper bound $\beta =2.0$, and the number of style transfer images $N=20$. For PGN, we set the number of examples $N=20$, the balanced coefficient $\omega =0.5$, and the upper bound of the neighborhood size $\varsigma =3·ϵ$. For BSR, we use its official parameters. For our proposed SLRM, we set the mixed coefficient $\delta =0.5$ and the replacement ratio $q=0.25$.

Evaluation Criterion. We use ASR to evaluate the performance of attacks. An attack is considered successful if the black-box target model $f_{\mathrm{target}}$ misclassifies the adversarial example $x^{adv}$, i.e., the predicted label of $x^{adv}$ differs from the ground-truth label y of the original benign image x. ASR can be expressed as

$$\mathrm{ASR}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\mathbb{I}[f_{\mathrm{target}}\left(x_i^{adv}\right)\ne y_i],$$

where N is the total number of images, $x_i^{adv}$ is the adversarial example generated from the i-th benign image $x_i$ using a surrogate classifier f, $y_i$ is the ground-truth label of $x_i$, and $\mathbb{I}(·)$ is the indicator function, which equals 1 if the condition is true and 0 otherwise.

4.2. Attack a Single Model

In this subsection, we examine the attack success rates of four distinct baseline methods without (w/o) and with (w/) SLRM. Four commonly pretrained models (i.e., Res-50, VGG19, Mob-v2, and Inc-v3) are selected as surrogate models to generate adversarial examples; their success rates are then evaluated on the ten target models. As illustrated in Table 4, the initial row corresponds to the attack success rates (classification error rate) for clean images, while the subsequent rows represent the attack success rates for adversarial examples. Bold fonts are used to highlight superior results.

Table 4. Attack success rates (%) of SLRM under the single-model setting. The adversaries are generated using Res-50, VGG19, Mob-v2, and Inc-v3, respectively. * indicates the white-box model.

Model	Attacks	SLRM	CNNs						Transformers
			Res-50	VGG19	Mob-v2	Inc-v3	ResX101	ConvNeXt	ViT-B	DeiT-S	Efficient	RVT-T
-	Clean	-	5.5	14.1	14.3	4.9	6.5	3.4	6.5	7.9	5.1	7.5
Res-50	DIM	w/o	99.4 *	84.2	77.9	56.6	80.0	57.8	40.2	45.9	63.5	67.1
		w/	99.8 *	90.1	88.0	72.9	88.7	71.2	52.8	59.7	76.5	75.9
	TIM	w/o	100 *	80.3	77.2	53.8	74.5	47.2	39.4	42.5	54.7	54.2
		w/	100 *	86.2	84.1	67.5	85.3	57.9	51.1	53.6	68.3	65.7
	SIM	w/o	100 *	87.1	86.8	50.7	87.2	64.9	47.5	50.1	72.6	70.0
		w/	100 *	92.8	93.2	55.8	93.8	79.3	64.5	66.9	85.8	85.2
	Admix	w/o	100 *	90.5	89.3	53.2	91.4	73.4	57.7	58.5	77.3	77.9
		w/	100 *	95.3	94.8	59.1	95.1	82.5	67.3	69.5	86.3	86.2
VGG19	DIM	w/o	61.9	99.8 *	81.6	48.9	58.1	48.6	26.4	33.0	47.3	42.2
		w/	71.1	100 *	87.5	63.1	67.4	56.0	36.2	39.8	57.2	51.9
	TIM	w/o	61.5	99.8 *	80.6	52.9	57.9	40.7	30.4	32.8	44.0	38.5
		w/	69.5	100 *	87.3	65.8	65.4	48.2	39.2	41.7	54.4	46.9
	SIM	w/o	70.1	100 *	87.2	39.8	65.5	46.3	28.3	34.6	51.9	47.8
		w/	81.6	100 *	92.8	45.9	74.2	59.2	40.1	47.2	67.8	61.3
	Admix	w/o	82.7	100 *	92.7	43.9	75.4	57.9	36.4	38.0	58.4	53.5
		w/	85.6	100 *	94.9	48.6	80.3	64.1	42.8	47.2	69.5	65.2
Mob-v2	DIM	w/o	53.8	81.6	100 *	41.5	56.8	34.9	30.1	32.9	50.5	50.6
		w/	64.2	88.3	100 *	56.1	68.2	46.3	41.0	46.1	63.8	60.2
	TIM	w/o	54.6	80.4	100 *	48.5	60.4	33.4	34.5	37.5	51.1	49.0
		w/	66.8	88.6	100 *	63.1	71.5	42.3	48.0	49.1	63.2	59.5
	SIM	w/o	62.7	85.9	100 *	34.5	66.8	40.5	34.3	39.8	62.4	57.8
		w/	76.4	93.5	100 *	39.9	79.4	53.5	50.3	54.6	77.2	73.9
	Admix	w/o	73.2	89.1	100 *	40.7	74.2	47.5	43.4	42.8	66.9	63.4
		w/	78.2	93.5	100 *	41.4	80.0	54.4	51.2	54.5	76.6	74.3
Inc-v3	DIM	w/o	42.7	63.0	61.5	99.4 *	46.6	27.9	21.9	26.9	40.3	36.2
		w/	54.7	71.5	70.3	99.8 *	57.3	36.1	31.8	35.2	50.4	45.6
	TIM	w/o	38.5	60.8	60.9	100 *	42.7	21.7	21.8	23.9	32.1	27.5
		w/	47.1	69.5	68.4	100 *	50.2	28.9	31.9	32.2	41.2	35.9
	SIM	w/o	50.1	67.1	67.1	100 *	51.9	29.6	28.4	30.6	44.9	41.4
		w/	61.3	76.5	75.7	100 *	63.8	41.6	39.3	41.0	57.0	54.1
	Admix	w/o	51.5	69.8	68.1	100 *	55.5	36.0	35.1	34.3	48.3	46.6
		w/	63.1	78.0	78.1	100 *	66.3	44.8	43.3	42.1	59.6	56.2

Bold values indicate better results; ties are also bolded.

A thorough examination of the data in Table 4 reveals that the baseline methods consistently attain remarkably high ASRs in the white-box setting. Our method attains attack success rates equal to or higher than those of the baseline methods on all these white-box models. However, the attack success rates of the baseline methods drop significantly on black-box models that differ substantially from the surrogate model. For instance, the SIM approach achieves a $100\%$ success rate on the white-box model Mob-v2, but only $34.3\%$ on the black-box model ViT-B. A similar pattern emerges when attacking Efficient using VGG19 as a surrogate model, where the various baselines achieve $47.3\%$ (DIM), $44.0\%$ (TIM), $51.9\%$ (SIM), and $58.4\%$ (Admix), respectively. In contrast, integrating SLRM with the baselines raises ASR to $57.2\%$ (DIM), $54.4\%$ (TIM), $67.8\%$ (SIM), and $69.5\%$ (Admix). These findings demonstrate that SLRM can remarkably enhance the attack success rates of baseline methods on black-box models, particularly on black-box models that exhibit substantial disparities from surrogate models.

4.3. Attack with SOTA Methods

We next evaluate the attack performance of SLRM when integrated with the state-of-the-art gradient-based method PGN, the input transformation-based method STM, and BSR. We also construct a stronger baseline, TDM, by combining the baseline methods DIM and TIM. As demonstrated in Table 5, these stronger baseline methods exhibit a substantial enhancement in their attack performance compared to the standard baseline methods. For instance, the four new baselines achieve average success rates of $70.5\%$ (TDM), $85.3\%$ (PGN), $83.7\%$ (STM), and $88.2\%$ (BSR) across the ten models. Specifically, when attacking ResX101, the four baselines achieve $85.1\%$, $92.1\%$, $90.7\%$, and $94.3\%$, respectively. Our SLRM further enhances the attack performance of these baselines, achieving $92.5\%$, $97.4\%$, $97.5\%$, and $98.3\%$, respectively, on ResX101. On average, SLRM enhances the ASR of the baseline methods on the ten models by $9.7\%$ (TDM), $5.7\%$ (PGN), $7.6\%$ (STM), and $5.1\%$ (BSR). These findings offer evidence that SLRM can be effectively integrated with advanced methods, thereby achieving enhanced attack performance.

Table 5. Attack success rates (%) of SLRM when integrated with SOTA attacks. The adversarial examples are generated on Res-50. * indicates the white-box model.

Attacks	SLRM	CNNs						Transformers
		Res-50	VGG19	Mob-v2	Inc-v3	ResX101	ConvNeXt	ViT-B	DeiT-S	Efficient	RVT-T
TDM	w/o	99.7 *	85.6	83.5	72.5	85.1	60.2	52.1	55.2	72.1	68.4
	w/	99.7 *	92.5	92.0	82.8	92.5	70.5	64.2	66.0	82.4	79.2
PGN	w/o	99.7 *	93.9	93.4	71.2	92.1	83.2	78.2	78.5	88.5	88.6
	w/	99.8 *	97.3	97.0	79.6	97.4	88.8	85.8	84.5	94.7	94.1
STM	w/o	99.5 *	93.8	90.6	74.5	90.7	80.5	75.3	77.5	85.1	85.0
	w/	99.9 *	96.9	96.8	82.8	97.5	88.4	86.2	83.3	95.6	94.2
BSR	w/o	100 *	94.4	93.2	82.5	94.3	84.2	82.4	82.6	90.2	89.9
	w/	100 *	95.6	96.9	89.7	98.3	90.2	89.2	89.8	94.5	95.4

Bold values indicate better results; ties are also bolded.

4.4. Attack via Ensemble Models

Ensemble attacks that leverage multiple surrogate models have been shown to enhance the transferability of adversarial examples [56].

To further validate the effectiveness of SLRM with an ensemble of surrogate models, we adopt the ensemble method in [10], which fuses the label confidence outputs of different models. Specifically, the outputs of the four standardly trained models (i.e., Res-50, VGG19, Mob-v2, and Inc-v3) are averaged, and the gradients with respect to the input are computed from the fused output to perform the ensemble attack. The results are presented in Table 6, which reports the ASR of the adversarial examples generated by various attacks via the ensemble model.

Table 6. Attack success rates (%) of SLRM under the ensemble-model setting. The adversaries are generated on ensemble models, i.e., Res-50, VGG19, Mob-v2, and Inc-v3. * indicates the white-box model.

Attacks	SLRM	CNNs						Transformers
		Res-50	VGG19	Mob-v2	Inc-v3	ResX101	ConvNeXt	ViT-B	DeiT-S	Efficient	RVT-T
DIM	w/o	98.5 *	100 *	100 *	99.0 *	90.5	80.2	60.3	64.5	84.6	81.4
	w/	98.8 *	100 *	100 *	99.1 *	96.3	88.1	73.6	78.0	92.7	90.0
TIM	w/o	99.1 *	100 *	100 *	99.4 *	91.1	78.1	63.1	66.0	82.1	77.1
	w/	99.1 *	100 *	100 *	99.3 *	97.1	87.5	83.1	82.1	93.9	91.5
SIM	w/o	99.2 *	100 *	100 *	99.6 *	94.3	83.3	66.2	69.6	89.5	86.8
	w/	99.6 *	100 *	100 *	99.9 *	98.2	93.5	83.6	85.3	96.1	95.3
Admix	w/o	99.4 *	100 *	100 *	99.9 *	96.2	89.7	75.0	75.2	92.4	91.4
	w/	99.5 *	100 *	100 *	99.9 *	98.5	94.0	84.3	86.4	96.3	94.6

Bold values indicate better results; ties are also bolded.

As shown in Table 6, all attack methods achieve extremely high ASRs in the white-box case. Compared with the results in Table 4, all methods attain significantly higher ASRs on the CNN-based black-box models. Similarly, relative to the Transformer-based results in Table 4, the ASRs of these baseline attacks are also greatly improved. For instance, TIM achieves $63.1\%$ on ViT-B, $66.0\%$ on DeiT-S, $82.1\%$ on Efficient, and $77.1\%$ on RVT-T, respectively. The proposed SLRM can further improve the ASRs of the baseline methods on Transformer-based models. For example, when SIM is used as the baseline to attack the four Transformer-based black-box models, its success rates are $66.2\%$, $69.6\%$, $89.5\%$, and $86.8\%$, respectively. When combined with SLRM, its ASR improves significantly across all four models: ViT-B ($66.2\%\to 83.6\%$), DeiT-S ($69.6\%\to 85.3\%$), Efficient ($89.5\%\to 96.1\%$), and RVT-T ($86.8\%\to 95.3\%$). Similarly, using DIM as the baseline, the ASR improves to ViT-B ($60.3\%\to 73.6\%$), DeiT-S ($64.5\%\to 78.0\%$), Efficient ($84.6\%\to 92.7\%$), and RVT-T ($81.4\%\to 90.0\%$), respectively. These experimental results demonstrate the efficacy of the SLRM in leveraging the ensemble gradient method, thereby enhancing the transferability of the baseline method. These results underscore the compatibility of the proposed method.

4.5. Attack Defense Models

In practical applications, deep learning models typically employ defensive strategies to counter potential attacks. Consequently, assessing the efficacy of an attack method on a defense model can, to a certain extent, reflect its performance in actual scenarios. To further validate the effectiveness of SLRM, we evaluated it against nine widely used defense methods, including the NIPS competition winner HGD [37], four input preprocessing defense methods (i.e., RS [40], ComDefend [38], NRP [39], and DiffPure [41]), and four adversarial training models (i.e., Inc-v3_adv [55], Inc-v3_ens3, Inc-v3_ens4, and IncRes-v2ens [36]). For ComDefend, NRP, and DiffPure, Mob-v2 served as the evaluation model, while for the other defense methods, we employed the official models provided by the respective authors.

Adversarial training methods have been demonstrated to serve as a straightforward and efficacious defense mechanism. Table 7 reports the ASRs of the four baseline methods against the four adversarially trained models, using Res-50 as the surrogate model. The results in Table 7 demonstrate that compared with the ASRs on the undefended models in Table 4, the rates of all four attacks (DIM, TIM, SIM, and Admix) are significantly reduced. Furthermore, when SLRM is employed in conjunction with these methods, enhancements of $10.7\%$ (DIM), $10.9\%$ (TIM), $8.9\%$ (SIM), and $6.6\%$ (Admix) are observed on the four adversarial training models. This suggests that our SLRM method can enhance the robustness of the baseline method against black-box adversarial training models. Beyond adversarial training methods, image denoising is also an effective defense mechanism. NRP and DiffPure are two powerful denoising-based defenses against adversarial examples, and they have been shown to be effective against transfer attack methods. Integrating SLRM with the baseline methods has been shown to enhance their resilience, thereby enhancing the robustness of the attacks against defenses. For instance, when employing DiffPure as a defense method, SLRM enhances the baseline success rate by $7.9\%$ (DIM), $6.3\%$ (TIM), $7.7\%$ (SIM), and $8.9\%$ (Admix), respectively. Furthermore, the results in this subsection demonstrates the effectiveness of SLRM in attacking other defense methods. For instance, when attacking ComDefend, the SLRM method enhances the baseline attack by $8.9\%$ (DIM), $10.8\%$ (TIM), $8.9\%$ (SIM), and $8.4\%$ (Admix), respectively. This finding underscores the efficacy of SLRM in enhancing the resilience of diverse baseline attacks against defense models.

Table 7. Attack success rates (%) of SLRM under nine defense models. The adversarial examples are generated on Res-50.

Attacks	SLRM	HGD	RS	ComDefend	NRP	DiffPure	Inc-v3adv	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
DIM	w/o	14.0	24.6	40.8	18.9	13.6	24.9	22.8	21.7	14.3
	w/	19.7	32.5	49.7	27.8	21.5	37.4	34.2	32.3	22.5
TIM	w/o	18.8	25.3	37.3	18.5	14.3	34.2	33.9	32.2	25.3
	w/	24.6	30.2	48.1	25.2	20.6	44.3	45.9	44.1	34.7
SIM	w/o	20.2	29.2	50.8	25.7	19.8	20.6	22.2	22.1	16.0
	w/	26.8	34.0	59.7	32.0	27.5	28.1	30.7	33.0	24.5
Admix	w/o	22.7	30.0	53.6	26.1	19.5	25.0	26.2	28.0	20.2
	w/	29.4	35.4	62.0	33.4	28.4	30.7	32.2	34.1	28.9

Bold values indicate better results.

4.6. Visualization of the Replica

In Figure 2, the first column represents the original images, while the remaining four columns illustrate replica images generated by four attack methods: SIM, Admix, STM, and SLRM. For SIM and Admix, the visualized replicas are generated when the number of replicas $m_1=2$. For STM, the visualizations correspond to a sample size $N=2$. We observe that SIM and Admix produce substantial differences between the replica images and the original images due to their pixel-wise division operations. STM employs a style transformer to obtain diverse style information, leading to the appearance of pronounced color blocks in its replica images. By contrast, SLRM utilizes an extractor to capture features of the original image and reconstructs them using a reconstructor, thereby preserving semantic similarity between the replicas and the original images. This ensures the accuracy of the gradient computations.

Figure 2. Visualization of original images and replicas. The surrogate model is Inc-v3.

Figure 3 presents the similarity among the original image x, the adversarial example $x^{adv}$, and the replica $x^{scc}$ over different iteration rounds. Here, SIM is adopted as the baseline method for SLRM, and Inc-v3 is employed as the surrogate model. The results demonstrate that combining SIM with the proposed SLRM significantly enhances the similarity between the replicas and the original images. Compared to the results in Figure 2, SLRM not only maintains the semantic consistency of the replicas with the original images but also preserves pixel-level similarity. Furthermore, the semantic consistency between the adversarial examples and the replicas is also high, ensuring that the gradient variation in the adversarial examples stays stable during the mixed-gradient update process. Consequently, SLRM effectively improves the transferability of adversarial examples in existing input transformation-based attacks.

Figure 3. The similarity evaluation metric SSIM of our SLRM during the iterations. We use SIM as the baseline; the surrogate model is Inc-v3.

4.7. Statistical Validation

To assess whether the improvements achieved by SLRM are statistically significant rather than due to random variation, we conducted three complementary analyses on the paired per-image attack outcomes (success or failure) for all 1000 images across every source model, attack method, and target model: McNemar’s test, the exact binomial test, and bootstrap confidence intervals for the difference in attack success rate ($\Delta \mathrm{ASR}={\mathrm{ASR}}_{\mathrm{SLRM}}-{\mathrm{ASR}}_{\mathrm{baseline}}$). The complete results are reported in Appendix B.

Across nearly all configurations, the McNemar p-values are far below $0.001$, confirming highly significant changes in the paired outcomes. The exact binomial $95\%$ confidence intervals for the proportion of discordant pairs where SLRM succeeds and the baseline fails are substantially above $0.5$, indicating that SLRM is systematically superior on those images where the two methods disagree. Bootstrap $95\%$ confidence intervals for $\Delta$ASR are positive, with typical gains of 5–15 percentage points that are much larger than the associated standard errors; these intervals rarely include zero. The few non-significant cases (marked with asterisks in the appendix tables) are scattered across specific model–attack pairs and represent a very small minority of all settings. Together, these results confirm that the improvements offered by SLRM are both statistically significant and practically meaningful.

4.8. Ablation Studies

We perform a series of ablation experiments to examine the impact of the two hyperparameters, $\delta$ and q, in the proposed SLRM. We also investigate the impact of different replicas on the performance of SLRM. We utilize SIM as the baseline for all adversarial examples.

The mixed coefficient $\delta$. In Figure 4a, we examine the impact of the mixed coefficient $\delta$ on transferability, with a replacement ratio $q=0.25$ for the replica’s gradient. As $\delta$ increases, the transferability of the adversarial examples to these black-box models improves, reaching its peak at $\delta =0.5$. This suggests that the best performance is achieved when the gradients of the adversarial example $x^{adv}$ and the replica $x^{scc}$ are equally mixed. We thus set $\delta =0.5$ for our experiments.

Figure 4. Ablation study on the mixed coefficient $\delta$ and the replacement ratio q. (a) Mixed coefficient $\delta$. (b) Replacement ratio q.

The replacement ratio q. In Figure 4b, we investigate the effect of the gradient replacement ratio, denoted by the parameter q, on the ASR. In our approach, we perform gradient enhancement by replacing the low-gradient areas of the replica. As q increases, the transferability of the adversarial examples also improves, reaching its peak at $q=0.25$. This indicates that the gradient replacement ratio in the replica reaches an optimal value, and increasing q beyond that point may decrease the ASR by overly altering the gradients of the replica. We thus set $q=0.25$ in our experiments.

Strategies for Generating Replicas and Their Discrepancy over Iterations. We evaluate the effect of replicas from different strategies on our method in Figure 5. In Figure 5a, we show the ASR of the original clean image, the noisy image (i.e., adding random noise of size $16/255$ to the original image), and the replica generated by our SLRM. Compared to the noisy image, our SLRM better preserves the semantic consistency of the replica with the original image. In Figure 5b, we illustrate the effect on SLRM when different replicas are used. The result for SIM-Original reflects the attack success rate of the adversarial example when the original clean image is used as the replica (i.e., setting $x_0^{scc}=x$ in our Algorithm 1). The result for SIM-Noise corresponds to the setting $x_0^{scc}=\mathrm{Noise}$. The result for SIM-SLRM corresponds to the setting $x_0^{scc}=x^{\prime }$ (i.e., the result reported in this paper). The results show that using clean images as replicas does not improve the transferability of adversarial examples. Compared with our SLRM, randomly noisy images destroy semantic consistency and only marginally improve transferability. By contrast, SLRM significantly improves the transferability of adversarial examples, demonstrating that semantic consistency enhances transferability.

Figure 5. Ablation study on the replicas generated by different strategies. The surrogate model is Inc-v3. (a) Attack success rates (%) of the adversarial examples. (b) Attack success rates (%) of the adversarial examples.

We evaluate the classification accuracy of generated replicas across five models (i.e., Inc-v3 [48], Inc-v3_adv [55], Inc-v3_ens3, Inc-v3_ens4, and IncRes-v2ens [36]), as shown in Figure 6a. Figure 6b shows the attack success rates of adversarial examples generated by SIM (with Inc-v3 as the surrogate model) against the five models over successive iterations. The results reveal a gradual decline in semantic consistency between the augmented replicas and the original image during iterative optimization. Furthermore, the transferability of adversarial examples generated by SIM increases rapidly over the first five iterations and plateaus after the sixth iteration. This phenomenon suggests that the discrepancy between the replica images and original inputs gradually widens with more iterations. These inconsistent replicas are associated with reduced accuracy in gradient computation, which may limit further improvement in adversarial example transferability.

Figure 6. The classification accuracy (%) of the replicas generated by SIM and the attack success rate (%) of the adversarial examples at iteration t. The surrogate model is Inc-v3. (a) Classification accuracy (%) of the replicas. (b) Attack success rate (%) of the adversarial examples.

4.9. Quantitative Analysis of Low-Gradient Hypothesis

To visualize the attention areas of adversarial examples, we employ Grad-CAM++ [57]. Although Grad-CAM++ provides only an approximate localization of important regions and may miss fine-grained or distributed features, it serves as a useful heuristic guide to identify candidate regions for analysis. Since Grad-CAM++ utilizes gradient values as weighting coefficients during visualization, it effectively reflects the gradient magnitude distribution (warmer colors indicate higher gradients). The first row in Figure 7 displays visualizations of original images across three models. The second and third rows show visualizations of adversarial examples generated using Inc-v3 as the white-box surrogate model with MIM [10] and our proposed method (MIM+Ours), respectively. As shown, the low-gradient replacement strategy may amplify gradient responses in regions that previously exhibited low gradients, thereby potentially improving the attack success rate and transferability of adversarial examples.

Figure 7. Visualization of the attention maps for the Inc-v3, Inc-v4 [48] and Res-101 [2] models. We apply Grad-CAM++ [57] to visualize the attention maps of the benign image (1st row) and the adversarial example (2nd and 3rd rows) crafted by MIM [10] and our approach. The 1st row shows the attention maps of the benign image on the three models. The 2nd and 3rd rows show the attention maps of adversarial examples crafted on the surrogate model (Inc-v3) and used to attack the black-box model (Inc-v4 and Res-101).

5. Conclusions

We argue that most input transformation-based attacks tend to produce semantically inconsistent replicas. Gradient-based attacks have some low-gradient areas corresponding to the minimal perturbation areas in their adversarial examples. These factors may affect the transferability of adversarial examples. Thus, we propose a Semantic-Invariance and Low-Gradient Replacement Method (SLRM). Specifically, we first extract the semantic features of the original input with an extractor and reconstruct them into a semantically invariant replica. Then, we perform gradient enhancement by replacing the low gradients of the replica with the gradients of the adversarial examples. We use Grad-CAM++ to heuristically identify candidate regions for analysis, and we provide a theoretical analysis to support our hypothesis. Extensive experimental results show that SLRM can significantly improve the adversarial transferability of various existing attacks. Our work demonstrates the potential of semantic-invariant and low-gradient replacement in improving the transferability of adversarial examples, offering valuable insights for further exploration in the domain of adversarial attacks.

SLRM is designed for non-targeted, ℓ_∞-constrained, inaccessible black-box digital attacks. It does not address ℓ₂ or targeted attacks, nor does it evaluate on datasets other than ImageNet. While we demonstrate that SLRM improves transferability on both CNN and Transformer architectures, a detailed architectural sensitivity analysis (e.g., how the effect varies with self-attention vs. convolution) remains an open question for future investigation. Furthermore, our ablation study validates the effectiveness of the semantic replica compared to noise and original images. But, a rigorous isolation of semantic consistency from other implicit regularization effects introduced by the reconstruction process (e.g., smoothing or information bottleneck) remains an open challenge for future work. In particular, we do not conduct physical-world tests due to equipment limitations related to camera sensors, lighting variations, camera angles, and other factors. We acknowledge these limitations and plan to explore them in subsequent work.

This work is intended solely as a tool for evaluating and improving model robustness. By revealing vulnerabilities in current black-box defenses, SLRM provides actionable insights for developing more resilient architectures. We emphasize that adversarial attacks should only be deployed in authorized, ethical settings to strengthen, not compromise, AI security. The semantic-invariance and low-gradient replacement mechanisms introduced here may also inspire novel defense strategies, such as gradient regularization or attention-aware smoothing, that explicitly mitigate these attack pathways.