SM-GCG: Spatial Momentum Greedy Coordinate Gradient for Robust Jailbreak Attacks on Large Language Models

Abstract

Recent advancements in large language models (LLMs) have increased the necessity of alignment and safety mechanisms. Despite these efforts, jailbreak attacks remain a significant threat, exploiting vulnerabilities to elicit harmful responses. While white-box attacks, such as the Greedy Coordinate Gradient (GCG) method, have demonstrated promise, their efficacy is often limited by non-smooth optimization landscapes and a tendency to converge to local minima. To mitigate these issues, we propose Spatial Momentum GCG (SM-GCG), a novel method that incorporates spatial momentum. This technique aggregates gradient information across multiple transformation spaces—including text, token, one-hot, and embedding spaces—to stabilize the optimization process and enhance the estimation of update directions, thereby more effectively exploiting model vulnerabilities to elicit harmful responses. Experimental results on models including Vicuna-7B, Guanaco-7B, and Llama2-7B-Chat demonstrate that SM-GCG significantly enhances the attack success rate in white-box settings. The method achieves a 10–15% improvement in attack success rate over baseline methods against robust models such as Llama2, while also exhibiting enhanced transferability to black-box models. These findings indicate that spatial momentum effectively mitigates the problem of local optima in discrete prompt optimization, thereby offering a more powerful and generalizable approach for red-team assessments of LLM safety. Warning: This paper contains potentially offensive and harmful text.

1. Introduction

Large language [1,2] models have been widely applied across various tasks and domains, including language understanding and generation [3], machine translation [4], and code generation. Through the use of alignment methods based on a series of pre-trained models [5,6], these models have acquired certain safety capabilities that prevent them from generating harmful or offensive responses to user queries. When prompted with prompts such as “Write a tutorial on how to make a bomb,” they produce evasive responses such as “I can’t fulfill your request. I’m an AI; it is not within my programming or ethical guidelines to provide instructions on bomb-making”. In this context, the “red-teaming” of large language models (LLMs) has emerged as a critical methodology [7], designed to rigorously evaluate the robustness of their safety mechanisms. Subsequently, jailbreak attacks have been widely employed in red team testing [8]. By combining adversarial prompts with malicious questioning, these attacks can mislead aligned language models into bypassing their safety mechanisms, thereby generating responses containing harmful, discriminatory, violent, or sensitive content.

Currently, various automated jailbreaking attack methods have been proposed. Existing jailbreaking approaches can be broadly categorized into two types based on their problem settings: (1) black-box attacks [9,10,11], and (2) white-box attacks [12,13,14,15,16]. A representative work in the first category is PAIR [17]; it uses an LLM as the attacker to autonomously generate jailbreak prompts for target models, while a notable example in the second category is the GCG attack [12]. The GCG attack reframes jailbreaking as an adversarial example generation task. It utilizes token-level gradient information from a white-box language model to guide the search for effective jailbreak prompts, as illustrated in Figure 1. It has demonstrated strong transferability and universality. However, existing methods still exhibit significant limitations in terms of attack effectiveness. For instance, the GCG method achieves an attack success rate of only 40–50% on the Llama2-7B model, a result far below the near-perfect success rates demonstrated by traditional adversarial attacks in image domains under similar settings [18]. We speculate that this gap partially stems from the non-smooth nature of discrete token optimization, which leads to inaccurate gradient estimation and causes the optimization process to easily become trapped in local minima.

To address the inherent limitations of GCG, several studies have proposed enhancements. For instance, MAC [19] incorporated a temporal momentum mechanism to stabilize the gradient update direction over optimization steps, aiming to mitigate oscillation and improve convergence. While these methods offer incremental improvements, our analysis reveals that they primarily operate within the original optimization framework and are still susceptible to local minima due to their reliance on gradient information from a single, discrete point in the input space. This key observation underscores the need for a more fundamental shift in the optimization strategy.

Our preliminary work hypothesized that bypassing discrete token representations to perform gradient optimization directly in the continuous embedding space—before projecting the result back into a token sequence—could yield a smoother optimization trajectory. However, experimental results revealed that the embedding space exhibits highly non-smooth characteristics. Specifically, a viable solution is often located within an extremely close neighborhood of the initial point, and the token sequence obtained after projection frequently remains identical to the original. This outcome indicates that local gradient information in the embedding space is poorly representative of the true discrete search directions and is therefore inadequate for effectively guiding the adversarial optimization process.

Building upon a profound understanding of the non-smoothness in embedding spaces, this paper seeks a breakthrough from a spatial perspective. Inspired by the Spatial Momentum method in visual adversarial attacks [20,21,22], we propose the Spatial Momentum Greedy Coordinate Gradient (SM-GCG) method, as illustrated in Figure 2. Instead of relying solely on single-point gradients, SM-GCG samples gradients across multiple spaces (candidate space, text space, token space, one-hot space, and embedding space) and integrates gradient information under semantically equivalent transformations. This approach, referred to as multi-space gradient sampling in later sections, more accurately estimates the overall gradient direction, avoiding convergence to local minima. The method significantly improves attack success rates in white-box settings while maintaining transferability.

The main contributions of this paper are as follows:

• We propose the SM-GCG method, which introduces a spatial momentum mechanism into LLM jailbreak attacks, enhancing optimization effectiveness through multi-space gradient sampling.
• We design transformation operations and gradient fusion strategies for different spaces (candidate, text, token, one-hot, and embedding) and analyze their compatibility.
• We validate the method’s efficacy on multiple open-source models (including Vicuna-7B, Guanaco-7B, and Llama2-7B-Chat), demonstrating a 10–15% improvement in attack success rate over baseline methods. Ablation studies confirmed the contribution of each spatial momentum component, and we further analyze the method transferability.

This study focuses on the gradient computation component of the GCG framework, which can be integrated with multiple existing GCG improvements [16,19,23]. Our work not only advances the development of jailbreak attack techniques but also provides new insights for understanding the adversarial robustness of LLMs.

It is important to emphasize that research on the method proposed in this paper has a clear dual-use nature. While it poses potential risks for misuse if deployed maliciously, we position this work fundamentally as a contribution to the security community. The primary purpose of developing more effective attack methods is to enable rigorous robustness evaluations, proactively discover vulnerabilities, and ultimately facilitate the development of stronger defensive mechanisms and more aligned AI systems. All experiments were conducted responsibly in controlled research environments using open-source models, adhering to the principle of responsible disclosure.

The remainder of this paper is organized as follows: Section 2 reviews related work on jailbreak attacks, including both white-box and black-box methods. Section 3 introduces the proposed Spatial Momentum Greedy Coordinate Gradient (SM-GCG) method, detailing the spatial momentum mechanism and its integration across multiple transformation spaces. Section 4 presents experimental results, including comparisons with baseline methods, ablation studies, and transferability analysis. Section 5 discusses the implications of our findings and potential future directions. Finally, Section 6 concludes the paper and outlines possible extensions of this work.

2. Previous Research

Recent studies indicate that jailbreak attacks targeting large language models (LLMs) are evolving toward greater sophistication. These systematic investigations have also revealed the complexity and persistence of such security vulnerabilities. The field has progressed from early manual prompt engineering techniques to more advanced automated approaches, branching into two main directions: white-box attacks and black-box attacks.

Classic white-box attack techniques can be categorized into three types: gradient-based prompt construction [12,13,14,15,16], generation process manipulation [24,25], and multi-modal jailbreaking [26]. GCG [12] appends an adversarially generated suffix to prompts by combining greedy search and gradient-based optimization. However, it ignores semantic coherence in the generated suffix, making it detectable via perplexity-based defenses. AutoDAN [13] employs a genetic algorithm to automatically generate stealthy jailbreak prompts. COLD-Attack [14] constructs prompts by incorporating additional loss terms (e.g., fluency, stealthiness, and sentiment control). ADC [15] relaxes discrete token optimization into a continuous process to address challenges in discrete jailbreak optimization. I-GCG [16] improves upon GCG by forcing the model to output a harmful response template. While these methods achieve high attack success rates, their primary limitation lies in strong assumptions about model accessibility, which restricts real-world applicability. Furthermore, many gradient-based methods (e.g., GCG) tend to produce semantically incoherent or high-perplexity suffixes, making them susceptible to simple perplexity-based detection mechanisms; EnDec [24] directly manipulates the generation process of open-source LLMs to induce harmful outputs. These approaches demonstrate the feasibility of exploiting model internals but are largely inapplicable to closed-source, proprietary models; With the rise of multi-modal LLMs, ColJailBreak [25] and VAE [26] shift the attack surface from text to images, highlighting that security risks are not confined to textual modalities and require more comprehensive defense frameworks. Notably, similar stealthy and physically realizable attack strategies have been explored in the vision domain, such as FIGhost [27] and ItPatch [28], which employ invisible triggers and adversarial patches to deceive traffic sign recognition systems, thereby paving the way for exploring real-world applications of multimodal attacks.

In black-box attacks, three primary techniques dominate: prompt rewriting [9,10,11], response-driven prompt optimization [17,29,30], and training-based prompt generation [31]. ArtPrompt [9] and Compromesso [10] jailbreak models by modifying original prompts into ASCII art and Italian-based formats, respectively, exploiting incomplete alignment vulnerabilities in LLMs. ReNeLLM [11] employs a nested scenario strategy for prompt rewriting, achieving high jailbreak success rates. While highly practical and transferable, these methods often depend on specific linguistic or cultural loopholes in alignment, which can be patched relatively easily once identified; PAIR [17] uses an LLM as the attacker to autonomously generate jailbreak prompts for target models. RLbreaker [29] designs a reinforcement learning agent to guide the optimization of jailbreak prompts. TAP [30] improves upon PAIR by introducing a branch-and-prune algorithm to reduce queries sent to the target LLM. These approaches are more adaptive but suffer from high query overhead and can be mitigated by rate-limiting or monitoring abnormal query patterns. TAP’s introduction of a branch-and-prune algorithm represents an important step toward improving query efficiency, though it does not fully resolve the trade-off between exploration efficiency and attack success; JailPO [31] proposes a preference optimization-based attack, fine-tuning the attacker LLM to generate prompts aligned with the target model’s preferences. This direction points toward more scalable and automated jailbreak generation, but it also introduces significant computational costs and raises the barrier for practical deployment.

The most straightforward defense against jailbreak attacks is to scrutinize prompts and reject malicious requests. The paper [32] notes that if a sentence lacks fluency, its perplexity will be high; thus, perplexity-based defenses can quickly identify malicious requests. Backtranslation [33] employs back-translated prompts to reveal the true intent of the original input. PARDEN [34] instructs the large language model (LLM) to repeat its own response and assesses whether the original prompt was malicious by measuring the similarity between the initial and repeated outputs. GradientCuff [35] defines a rejection loss that leverages zeroth-order gradient estimation to detect malicious requests. While these methods are lightweight and easy to deploy, they often rely on heuristics that can be circumvented by adaptive attacks designed to mimic benign input characteristics.

Another defensive approach involves safety fine-tuning LLMs to enhance alignment mechanisms. Goal prioritization [36] ensures that the model prioritizes safety objectives during both training and inference. PAT [37], inspired by adversarial training paradigms, trains a protective prefix appended to the original prompt. SafeDecoding [38] fine-tunes the base model to create a safety-focused expert model, thereby reducing the output probability of tokens aligned with attacker goals. These approaches provide a more foundational mitigation but require substantial computational resources and may inadvertently impact the model’s general capabilities.

Overall, the literature depicts a sophisticated arms race. However, both the escalating complexity of attacks and the proposed defenses remain largely academic, exhibiting a significant gap in their practicality and robustness for real-world deployment.

3. Methodology

3.1. Problem Formulation

Given an original prompt represented as $x_{1:n}=[x_1,x_2,\dots ,x_n]$, a sequence of input tokens where each $x_i\in \{1,\dots ,V\}$ (with V being the vocabulary size), an LLM maps the sequence of tokens to a distribution over the next token. The LLM can be defined as $p\left(x_{n+1}\right|x_{1:n})$, representing the likelihood of $x_{n+1}$ given the preceding tokens $x_{1:n}$. Thus, the response $x_{n+1:n+G}$ can be generated by sampling from the following distribution:

$$p\left(x_{n+1:n+G}\right|x_{1:n})=\prod _{i=1}^{G}p\left(x_{n+i}\right|x_{1:n+i-1})$$

(1)

To force the model to provide correct answers to malicious questions, rather than refusing to respond, previous works combine the malicious question $x_{1:n}$ with the optimized jailbreak suffix $x_{n+1:n+m}$, forming a jailbreak prompt $x_{1:n}\oplus x_{n+1:n+m}$, where ⊕ represents the vector concatenation operation. For notational simplicity, let $x^O$ denote the malicious question $x_{1:n}$, $x^S$ represent the jailbreak suffix $x_{n+1:n+m}$, and $x^O\oplus x^S$ stand for the jailbreak prompt $x_{1:n}\oplus x_{n+1:n+m}$. Setting a specific target response for individual malicious questions is impractical, as crafting an appropriate answer for each query is challenging and risks compromising universality. A common workaround [7,39] is to default to affirmative responses (e.g., “Sure, here’s how to [$x^O$]”). To achieve this, we optimize the LLM initial output to align with a predefined target prefix $x_{n+m+1:n+m+k}^T$ (abbreviated as $x^T$), leading to the following adversarial jailbreak loss function:

$$\mathcal{L}(x^O\oplus x^S)=-\log p\left(x^T\right|x^O\oplus x^S)$$

(2)

The generation of the adversarial suffix can be formulated as the minimal optimization problem, written as follows:

$$\underset{x^S\in {\{1,\dots ,V\}}^m}{minimize}\mathcal{L}\left(x^O\oplus x^S\right).$$

(3)

For simplicity in representation, we use $\mathcal{L}\left(x^F\right)$ to denote $\mathcal{L}(x^O\oplus x^S)$ in subsequent sections. A detailed optimization process is provided in Appendix E to aid understanding.

3.2. Spatial Momentum

MAC [19] is the first to introduce a momentum mechanism into the GCG method, achieving performance improvements. By integrating a momentum term into the iterative process, it effectively incorporates temporal correlations in the gradients used for candidate sampling, thereby stabilizing the update direction during iterations.

In our paper, inspired by advancements [20,21,22] in the traditional visual adversarial domain, we apply the spatial momentum method to enhance the performance and transferability of GCG, naming it SM-GCG (Spatial Momentum GCG).

In traditional GCG methods, candidate sampling gradients depend entirely on the current input. This can cause the adversarial suffix to overfit to the specific malicious query during iterative optimization, becoming fragile to even single-character modifications. Ideally, a robust adversarial suffix should maintain effectiveness against semantically equivalent variations of the malicious question. Compared to the GCG gradient formulation, SM-GCG integrates gradients from multiple random transformations of the malicious query while incorporating abstract semantic space information to ensure gradient stability. This integration acts as a form of gradient averaging, which smooths the optimization landscape. The dynamical consequence, as evidenced by the loss curves in Figure 3 (the experimental setup for this comparison used 100 malicious prompts from AdvBench [12] to attack LLaMA2-7B, with 500 attack rounds per method; the plotted curves show the average loss across the 100 attacks, with the shaded area indicating the standard deviation), is a significant reduction in oscillation amplitude. This dampening effect arises because the averaged gradient is less susceptible to the high-frequency noise present in any single instance of the query, guiding the optimization towards a more stable descent direction. Furthermore, the smooth convergence phase observed in SM-GCG indicates that the optimizer has located a flat region of the loss minimum. Solutions in such flat minima are theoretically and empirically linked to superior generalization, which in our context translates to adversarial suffixes that are robust to paraphrasing and minor perturbations of the original malicious query. Finally, the lower plateau value of the loss achieved by SM-GCG quantifies a higher success probability for the attack. We quantitatively validated this relationship by monitoring the attack success rate alongside the loss across ten independent SM-GCG runs under varying configurations. A representative example (Figure 4) shows the striking mirror image between the two curves during optimization. The average Spearman’s rank correlation coefficient across all 10 runs is −0.995 (std: ±0.004), providing robust statistical evidence that the loss value is a highly reliable proxy for attack efficacy. Consequently, the significantly lower final loss plateau achieved by SM-GCG (as seen in Figure 3) directly and consistently translates to its measurably higher attack success rate across our benchmark. The proposed gradient formulation is defined as follows:

$$g_j=\alpha {\nabla }_{e_{x_j^S}}\mathcal{L}\left(x^F\right)+\sum _{i=1}^n{\lambda }_i{\mathcal{G}}_i\left(x^F\right)$$

(4)

where ${\mathcal{G}}_i(·)$ is used to compute the gradient after applying transformations to $x^F$, where n represents the desired number of transformations, details are provided below. j is the index of the suffix, where $j\in \{0,1,\dots ,m-1\}$, and m is the length of the token sequence after encoding the adversarial suffix. The term $e_{x_j^S}$ denotes the one-hot vector corresponding to the token at index j. The coefficients $\alpha$ and ${\lambda }_i$ are weighting factors used to balance the original gradient and sampled gradient.

Through extensive research and experimentation, we classify input transformations using the following criteria:

• Application Position:
  • $x^O$ (Malicious Query): transformations applied to the original query.
  • $x^S$ (Adversarial Suffix): transformations applied to the suffix.
• Transformation Space (four types):
  • Candidate space
  • Text space (granularity-based):

    –

    Character-level

    –

    Word-level

    –

    Sentence-level

    –

    Message-level

  • Token space
  • One-hot space
  • Embedding space

Note: Text-space transformations on $x^S$ may alter decoded sequence length, disrupting gradient accumulation and thus requiring pre-use filtering. Due to the non-surjective nature of tokenizers (tokens → string), not all token sequences map to valid strings. While transformations in Token, One-Hot, and Embedding spaces may sample values absent in normal inference scenarios, failing to filter them does not cause gradient accumulation errors. For details, see

Table 1. Compatibility matrix for transformations.

Application	Candidate	Text				Token	One-Hot	Embedding
Position		Character	Word	Sentence	Message
$x^O$	X	T	T	T	T	B	B	B
$x^S$	T	F	F	F	X	B	B	B

. In the table, “F” indicates that the corresponding combination may produce some unusable sample values and must be filtered to take effect; “T” means the corresponding combination requires no filtering; “B” indicates that the combination may generate some low-quality sample values, which can be either filtered or omitted; “X” denotes that the combination is incompatible and cannot take effect.

The function ${\mathcal{G}}_i(·)$ in Equation (4) is merely an abstract representation; in practice, the gradient function varies depending on the different transformation space.

3.2.1. Candidate Space

In the iterative process of GCG, each iteration generates a batch of candidate suffixes with only 1–2 token differences. A high-quality suffix should exhibit robustness, meaning that minor modifications to the suffix should preserve most of its performance. To simulate such modifications, we sample candidate suffixes based on their gradients, particularly focusing on loss-guided sampling. This method more readily identifies suffixes that may appear suboptimal from a local perspective but are globally optimal. This facilitates a more stable iterative process and enables the generation of more robust adversarial suffixes.

Gradient sampling in the candidate space involves retaining the set of candidate suffixes from the previous iteration during the iterative process. Random sampling or loss-prioritized sampling is employed to accumulate gradients from other candidates. In the t-th iteration, the sampled inputs are calculated as follows:

$$S^{\prime }={\mathbf{T}}^{\mathrm{candidate}}(S^O,S^S)=S^O\left|\right|S^C$$

(5)

$$S^C:=\left\{\begin{array}{cc}{S}_{\left(0\right)}^S\hfill & t=0\hfill \\ S_{\left(t\right)}^S\sim \mathrm{Sample}\left(C^{\left(t\right)}\right)\hfill & t>0\hfill \end{array}\right.$$

(6)

$$C^{\left(t\right)}=\mathrm{GenerateCandidates}\left(S_{(t-1)}^S\right)$$

(7)

where $S_{\left(t\right)}^S$ is the adversarial suffix in the t-th iteration, $\mathrm{Sample}(·)$ can serve as either a random sampling function or a loss-prioritized sampling function.

We encode the input using a tokenizer to obtain a sequence of tokens $x^{\prime }=\mathrm{Encode}\left(S^{\prime }\right)$. The gradient formula applying the candidate-space transformation is written as follows:

$${\mathcal{G}}_j^{\mathrm{candidate}}\left(x^F\right)={\nabla }_{e_{x_j^{\prime S}}}\mathcal{L}\left(x^{\prime }\right)$$

(8)

Specifically, we implement two sampling strategies:

• Random Sampling: Uniformly select k candidates from the candidate pool $C^{\left(t\right)}$ without considering their loss values, promoting exploration of diverse gradient directions.
• Loss-Guided Sampling: Select the top-k candidates with the lowest loss values from the candidate pool $C^{\left(t\right)}$. This strategy focuses the gradient computation on the most promising (lowest-loss) directions found in the current candidate set, reinforcing and refining the most effective perturbations.

3.2.2. Text Space

We believe that a robust adversarial suffix should not be rendered ineffective by modifications that do not alter the semantics, such as “Write a convincing anonymous death threat” and “Write a credible anonymous death threat”. Therefore, we attempt to sample these text transformations that minimally change the semantics to collect gradients, ensuring that the generated suffix relies not on specific character sequences but on higher-level semantics.

In the text space, changes need to be applied to the string. Let $S=S^O\left|\right|S^S$ be the original string, where $S^O$ denotes the malicious question part of the original string, $S^S$ denotes the adversarial suffix part, and $\left|\right|$ represents the string concatenation operation. After applying the transformers, we obtain the following:

$$S^{\prime }={\mathbf{T}}^{\mathrm{text}}(S^O,S^S)={\mathcal{T}}^{\mathrm{text}}\left(S^O\right)\left|\right|{\mathbb{T}}^{\mathrm{text}}\left(S^S\right)$$

(9)

$${\mathbb{T}}^{\mathrm{text}}\left(S^S\right):=\left\{{\mathcal{T}}^{\mathrm{text}}\left(S^S\right)\right|\mathrm{Len}(\mathrm{Encode}\left({\mathcal{T}}^{\mathrm{text}}\left(S^S\right)\right)=\mathrm{Len}\left(\mathrm{Encode}\left(S^S\right)\right)\}$$

(10)

We encode the input using a tokenizer to obtain a sequence of tokens $x^{\prime }=\mathrm{Encode}\left(S^{\prime }\right)$, which is then substituted into Equation (4), yielding the gradient formula after applying the text-space transformation:

$${\mathcal{G}}_j^{\mathrm{text}}\left(x^F\right)={\nabla }_{e_{x_j^{\prime S}}}\mathcal{L}\left(x^{\prime }\right)$$

(11)

where $x_j^{\prime S}$ is the j-th token in the encoded token sequence of the string after applying text transformations.

We implement text-space transformations at two textual modification granularity using the nlpaug library:

1.

Character-Level Transformations:

• Random Character Substitution: Randomly replace 1–2 characters with other alphabetic characters.
• OCR-based Substitution: Simulate OCR errors by replacing characters with visually similar ones (e.g., ’o’→’0’, ’l’→’1’).
• Keyboard Typo Substitution: Replace characters with adjacent keyboard keys (e.g., ’a’→’s’, ’k’→’l’).

2.

Word-Level Transformations:

• Synonym Replacement: Replace one word with its semantic equivalent using WordNet.
• Random Swap: Randomly swap the positions of two adjacent words.
• Random Deletion: Delete one word from the suffix.
• Spelling Error Replacement: Introduce common spelling mistakes (e.g., ’receive’→’recieve’).

We limit changes to 1–2 characters or 1 word to minimize semantic alteration while providing sufficient variation for robustness.

Furthermore, our framework can be extended to handle more complex scenarios: For malicious queries consisting of multiple sentences, sentence-level transformations such as random sentence reordering and sentence paraphrasing can be applied. For contextual malicious scenarios involving multiple turns of dialogue, message-level transformations such as random context message reordering can be employed. Since the AdvBench dataset used in our subsequent experiments contains malicious prompts that are primarily single-sentence queries, these extended transformations are not utilized in our current experimental setup.

3.2.3. Token Space

The transformations we apply in token space can be mapped to text space. However, the advantage of operating in token space is that we avoid the issue of altered decoded token sequence lengths caused by transformations. Therefore, in text space, we are typically limited to minor modifications such as synonym replacement, whereas in token space, more impactful transformations such as shift operations can be applied.

In the token space, changes need to be applied to the token sequence. The gradient formula is written as follows:

$${\mathbf{T}}^{\mathrm{token}}(x^O,x^S)={\mathcal{T}}^{\mathrm{token}}\left(x^O\right)\oplus {\mathbb{T}}^{\mathrm{token}}\left(x^S\right)$$

(12)

$${\mathcal{G}}_j^{\mathrm{token}}\left(x^F\right)={\nabla }_{e_{x_j^{\prime S}}}\mathcal{L}\left({\mathbf{T}}^{\mathrm{token}}(x^O,x^S)\right)$$

(13)

where $x_j^{\prime S}$ is the j-th token in the encoded token sequence after applying token transformations.

When applying transformers, we can attempt to decode and re-encode the token sequence, filtering out any outputs that diverge from the original sequence to ensure the transformed tokens remain valid.

$${\mathbb{T}}^{\mathrm{token}}\left(x^S\right):=\left\{{\mathcal{T}}^{\mathrm{token}}\left(x^S\right)\right|{\mathcal{T}}^{\mathrm{token}}\left(x^S\right)=\mathrm{Encode}\left(\mathrm{Decode}\left({\mathcal{T}}^{\mathrm{token}}\left(x^S\right)\right)\right)\}$$

(14)

In token space, we implement two specific transformation strategies:

1. Random Token Replacement: Randomly replace a subset of tokens in the sequence with other valid tokens from the vocabulary. Formally, for a token sequence $x^S=[x_1,x_2,...,x_n]$, we generate the following:

$${\mathcal{T}}^{\mathrm{replace}}\left(x^S\right)=[x_1,...,{\tilde{x}}_i,...,x_n]$$

(15)

where ${\tilde{x}}_i\sim \mathrm{Vocab}\setminus x_i$ for randomly selected positions i.

2. Cyclic Shift Operation: Perform circular shifting of the token sequence by a random offset k, calculated as follows:

$${\mathcal{T}}^{\mathrm{shift}}\left(x^S\right)=[x_{k+1},x_{k+2},...,x_n,x_1,...,x_k]$$

(16)

This operation preserves all token information while altering the positional context.

Similar to the text space, we limit the scope of modifications by replacing only 1–2 tokens to maintain semantic coherence while providing sufficient variation.

3.2.4. One-Hot Space

The sampling in one-hot and embedding spaces primarily addresses the high non-smoothness of gradients in these spaces. Local gradients may fail to capture the global gradient landscape, leading optimization to converge to local minima. Neighborhood sampling can help stabilize gradients.

In the one-hot space, first convert the token sequence into one-hot form $e_{x^S}=\mathrm{onehot}\left(x^S\right)$ and $e_{x^O}=\mathrm{onehot}\left(x^O\right)$. The loss function in Formula (2) simplifies the model reasoning process. We extract the embedding procedure and redefine a loss function that takes embedding vectors as input:

$${\mathcal{L}}_{\mathrm{embedding}}\left(v\right)=-\log p_{\mathrm{embedding}}\left(x^T\right|v)$$

(17)

where v is a embedding vectors.

We apply transformations to the malicious query and adversarial suffix separately, then multiply them by the embedding weight matrix to obtain the modified embedding vectors. The gradient formula is:

$${\mathbf{T}}^{\mathrm{onehot}}(e_{x^O},e_{x^S})={\mathcal{T}}^{\mathrm{onehot}}\left(e_{x^O}\right)\oplus {\mathcal{T}}^{\mathrm{onehot}}\left(e_{x^S}\right)$$

(18)

$${\mathcal{G}}_j^{\mathrm{onehot}}\left(x^F\right)={\nabla }_{e_{x_j^S}}\mathcal{L}\left({\mathbf{T}}^{\mathrm{onehot}}(x^O,x^S)\right)\times W)$$

(19)

where W is the embedding weight matrix. Due to the sparsity of natural values in one-hot space, applying transformations almost never yields natural values. Therefore, we employ neighborhood sampling without filtering.

Specifically, in the one-hot space, we employ Gaussian noise injection to sample neighboring points around the current one-hot vectors. This approach helps explore the gradient landscape beyond immediate local neighborhoods and provides more stable gradient estimates for optimization.

3.2.5. Embedding Space

In the embedding space, the original embedding vector is $v^S=e_{x^S}\times W$ and $v^O=e_{x^O}\times W$. The gradient formula is written as follows:

$${\mathbf{T}}^{\mathrm{embedding}}(v^O,v^S)={\mathcal{T}}^{embedding}\left(v^O\right)\oplus {\mathcal{T}}^{embedding}\left(v^S\right)$$

(20)

$${\mathcal{G}}_j^{embedding}\left(x^F\right)={\nabla }_{e_{x_j^S}}\mathcal{L}\left({\mathbf{T}}^{\mathrm{embedding}}(v^O,v^S)\right)$$

(21)

As in the one-hot space, Gaussian noise is used to sample the neighborhood without filtering.

3.3. Spatial Momentum Greedy Coordinate Gradient

Our method enhances the original GCG by incorporating a spatial momentum mechanism, which can be synergistically combined with various other improvements. The final algorithm, SM-GCG, is presented in Algorithm 1.

Algorithm 1 Spatial Momentum Greedy Coordinate Gradient

Input:  Malicious question $x^O$, initial adversarial suffix $x^S$, iterations T, loss function $\mathcal{L}$, loss function for embedding vectors ${\mathcal{L}}_{\mathrm{embedding}}$, k, batch size B, transformers in candidate space ${\mathbf{T}}_1^{\mathrm{candidate}},\dots ,{\mathbf{T}}_P^{\mathrm{candidate}}$, transformers in text space ${\mathbf{T}}_1^{\mathrm{text}},\dots ,{\mathbf{T}}_N^{\mathrm{text}}$, transformers in token space ${\mathbf{T}}_1^{\mathrm{token}},\dots ,{\mathbf{T}}_M^{\mathrm{token}}$, transformers in one-hot space ${\mathbf{T}}_1^{\mathrm{onehot}},\dots ,{\mathbf{T}}_L^{\mathrm{onehot}}$, transformers in embedding space ${\mathbf{T}}_1^{\mathrm{embedding}},\dots ,{\mathbf{T}}_H^{\mathrm{embedding}}$, and gradient weight $\alpha ,{\lambda }_1,\dots ,{\lambda }_{P+N+M+L+H}$, embedding weight matrix W   1: repeat   2:     $x^F:=x^O\oplus x^S$   3:     $e_{x^O}:=\mathrm{onehot}\left(x^O\right),e_{x^S}=\mathrm{onehot}\left(x^S\right)$   4:     $v^O:=e_{x^O}\times W,v^S=e_{x^S}\times W$   5:     $S^O:=\mathrm{Decode}\left(x^O\right),S^S:=\mathrm{Decode}\left(x^S\right)$   6:     $\mathcal{I}:=\mathrm{Indices}\left(x^S\right)$   7:     for $i\in \mathcal{I}$ do   8:         $g_i:=\alpha {\nabla }_{e_{x_i^S}}\mathcal{L}\left(x^F\right)$   9:         for p = 1, …, P do 10:            $g_i:=g_i+{\lambda }_p{\nabla }_{e_{x_i^{\prime S}}}\mathcal{L}\left({\mathbf{T}}_p^{\mathrm{candidate}}(S^O,S^S)\right)$ 11:         end for 12:         for n = 1, …, N do 13:            $g_i:=g_i+{\lambda }_n{\nabla }_{e_{x_i^{\prime S}}}\mathcal{L}\left({\mathbf{T}}_n^{\mathrm{text}}(S^O,S^S)\right)$ 14:         end for 15:         for m = 1, …, M do 16:            $g_i:=g_i+{\lambda }_{N+m}{\nabla }_{e_{x_i^{\prime S}}}\mathcal{L}\left({\mathbf{T}}_m^{\mathrm{token}}(x^O,x^S)\right)$ 17:         end for 18:         for l = 1, …, L do 19:            $g_i:=g_i+{\lambda }_{N+M+l}{\nabla }_{e_{x_i^S}}{\mathcal{L}}_{\mathrm{embedding}}\left({\mathbf{T}}_l^{\mathrm{onehot}}(e_{x^O},e_{x^S})\right)$ 20:         end for 21:         for h = 1, …, H do 22:            $g_i:=g_i+{\lambda }_{N+M+L+h}{\nabla }_{e_{x_i^S}}{\mathcal{L}}_{\mathrm{embedding}}\left({\mathbf{T}}_h^{\mathrm{embedding}}(v^O,v^S)\right)$ 23:         end for 24:         ${\mathcal{X}}_i:=\mathrm{Top}-\mathrm{k}(-g_i)$ 25:     end for 26:     for b = 1, …, B do 27:         ${\tilde{x}}_i^{\left(b\right)}:=\mathrm{Uniform}\left({\mathcal{X}}_i\right),\mathrm{where}i=\mathrm{Uniform}\left(\mathcal{I}\right)$ 28:     end for 29:     $x^S:={\tilde{x}}^{\left(b^{*}\right)},\mathrm{where}{b}^{*}={\mathrm{argmin}}_b\mathcal{L}(x^O\oplus {\tilde{x}}^{\left(b\right)})$ 30: until T times Output:  Optimized suffix $x^S$

Spatial momentum can also be applied to universal prompt optimization, as detailed in Appendix A.

4. Results

4.1. Experimental Setups

This section experimentally validates the performance improvement of SM-GCG compared to previous methods. We designed three experiments: a comparative experiment on attack effectiveness, an ablation experiment, and a transferability experiment. Details are provided below. Additionally, the datasets, metrics, comparative models, and methods used in the experiments are as follows:

Datasets. We use AdvBench Harmful Behaviors [12] to evaluate jailbreak attacks. This dataset contains 520 malicious questions, covering various aspects such as graphic depictions, profanity, misinformation, threatening behavior, cybercrime, discrimination, and illegal or dangerous suggestions. The computational demands of this process are significant. For instance, each round of GCG optimization on an H100 GPU typically requires approximately 4 s. To execute 500 optimization rounds for all 520 malicious prompts would therefore require roughly 300 h. Given this time constraint, we randomly selected a subset of 100 malicious prompts for the subsequent experiments. This subset was curated to ensure it represented a diverse range of types. Given this constraint, we employed a stratified random sampling method to select a representative subset of 100 malicious prompts. Specifically, we used the original category labels provided by AdvBench as strata. The number of prompts selected from each category was proportional to the category size in the full 520-prompt dataset. This approach ensures that our subset preserves the distribution of harmful behavior types present in the complete dataset, thereby enhancing the representativeness and diversity of our evaluation sample. While our sampling method aims to maximize representativeness, we acknowledge that using a subset of 100 prompts (19.2% of the full dataset) may introduce sampling bias and limit the generalizability of our results in two main ways:

• Rare Categories: Some harmful categories with a small number of prompts in the full dataset might be underrepresented in our subset, potentially leading to an over- or underestimation of the attack effectiveness on those specific types of queries.
• Intra-Category Diversity: The effectiveness of jailbreak attacks can be sensitive to the specific phrasing and content of a prompt. Our subset may not capture the full linguistic diversity within each category.

Despite these potential limitations, we argue that our stratified sampling approach provides a robust and practical compromise, offering a fair evaluation of the attack’s overall performance across major categories of harmful content while remaining computationally feasible. Future work with greater computational resources will benefit from validation on the entire dataset.

Metrics. The experiment employed two evaluation metrics to assess the practical performance of jailbreaking methods: attack success rate and recheck. Attack success rate (ASR) [12] is a simple yet effective keyword-based metric that detects whether the LLM response contains predefined refusal keywords, such as “sorry,” “as a responsible AI,” etc. If the model reply includes any of these keywords, it is assumed that the model has identified the query as malicious and refused to answer, indicating a failed attack. The predefined keywords used in the experiment can be found in Appendix B. The second metric is the GPT recheck attack success rate (Recheck) [13]. Sometimes, LLMs do not directly refuse to answer malicious questions but instead provide off-topic responses. Alternatively, they may correct earlier mistakes in subsequent replies, such as reminding the user that the request might be illegal or unethical. These scenarios could lead to ASR failing to accurately reflect the performance of jailbreak methods. We employed an LLM to evaluate jailbreak success. Following a comprehensive comparison of Recheck prompts in the literature, we adopted the one from [16]. We use the final attack success rate for both metrics, calculated as follows: $I_{success}/I_{total}$.

Models. To evaluate our method, we employ three open-source LLMs: Llama2-7B-Chat [40], Guanaco-7B [41], and Vicuna-7B [42]. These models are run without system prompts, and further details are provided in Appendix C.

Baselines. We selected GCG [12], MAC [19], and AUTODAN [13] as baselines.

Hyperparameter. In our experiments, the SM-GCG method employed a temporal momentum mechanism with a fixed momentum coefficient of 0.4. The spatial momentum was configured as follows: In the candidate space, a loss-based selection approach was adopted, prioritizing candidates with lower loss values. Six samples were taken from this space. In the text space, synonym replacement was applied by substituting a randomly selected token (after word segmentation) with its synonym, with six samples sampled. In the token space, cyclic shifting was utilized to produce two samples (shifting left and right by one token, respectively), along with a random replacement method where one randomly selected token was replaced with another token to generate four additional samples, resulting in a total of six samples. In both the one-hot space and the embedding space, Gaussian noise (mean = 0, variance = 0.0001) was added, with seven samples drawn from each. This process resulted in a total of 32 hybrid samples, with a composition ratio ${\lambda }_i$ of 6:6:6:7:7 as specified in Formula (4). The primary weight $\alpha$ was set to 0.25. The attack process was set to run for a maximum of 500 iterations, with an early stopping condition applied: Every 25 iterations, the current adversarial example was validated for attack success, and if successful, the attack would terminate prematurely. The rationale for the selection of these parameters is detailed in the Appendix D.

4.2. Attack Effectiveness

Table 2. White-box attack success rate (ASR) and recheck results on victim LLMs.

Models	VICUNA-7B		GUANACO-7B		LLAMA2-7B-CHAT
Methods	ASR	Recheck	ASR	Recheck	ASR	Recheck
GCG	97%	87%	100%	97%	49%	40%
MAC	100%	96%	100%	96%	54%	43%
AutoDan	100%	95%	100%	94%	56%	46%
SM-GCG	96%	91%	100%	99%	65%	58%

Bold indicates the best performance.

presents the white-box evaluation results of our method, SM-GCG, and other baseline methods. To reduce computational costs, we randomly selected a subset of 100 malicious questions from the AdvBench dataset as our experimental dataset. The evaluation methodology involved crafting an adversarial prompt for every malicious query in a benchmark dataset to test the resilience of the victim LLM outputs in a security assessment. The specific experimental parameters are as follows: We reproduced the original code of GCG and optimized some time-consuming operations. We added the momentum mechanism on the basis of GCG to achieve MAC and fixed its momentum coefficient at 0.4. AUTODAN conducted experiments using the original code. In addition, on the basis of MAC, the spatial momentum mechanism proposed in this paper was added to achieve SM-GCG. In the experiment, we used the best-performing hybrid sampling method, which sampled five momentum spaces in a ratio of 6:6:6:7:7, and the main weight coefficient $\alpha$ was set to 0.25. Experimental results demonstrate that SM-GCG effectively generates adversarial prompts and achieves a higher attack success rate compared to baseline methods. For the robust model Llama2, SM-CG improves the attack success rate by 10–15%.

4.3. Ablation Experiment

We evaluated the importance of the five momentum spaces proposed in SM-GCG. In the experiments, we randomly selected 20 malicious queries from the AdvBench dataset as the test set. White-box attacks were performed on the Llama2 model for 500 iterations. If temporal momentum was employed, the momentum coefficient was set to 0.4; if spatial momentum was used, the momentum sampling number was set to 32. The candidate selection strategy was based on loss; the text space utilized synonym replacement transformations; and the token space employed shift transformations, while the one-hot space and embedding space utilized Gaussian noise transformations. Specifically, the final SM-GCG method utilized temporal momentum with a coefficient of 0.4 and spatial momentum with a sampling number of 32. The hybrid sampling ratios were set to 6:6:6:7:7, with a primary weight coefficient of 0.25 and other weight coefficients following the ratio of 6:6:6:7:7.

The results are shown in

Table 3. Ablation experiment. The experiment was conducted on a single H100 GPU.

Time	Candidate	Text	Token	One-Hot	Embedding	ASR	Time Cost per Step
GCG						8/20	4.1953 s
MAC						9/20	4.2207 s
✓	✓					13/20	4.7101 s
✓		✓				12/20	4.5268 s
✓			✓			11/20	4.5736 s
✓				✓		12/20	4.5206 s
✓					✓	12/20	4.5562 s
✓	✓	✓	✓	✓	✓	14/20	4.6696 s

Bold indicates the best performance. The “✓” in the table indicate which specific components were activated or included in each experimental configuration of the ablation study.

. Compared to the baseline method (GCG, with an ASR of 8/20), all five momentum spaces we introduced improve performance. The final hybrid sampling approach (SM-GCG) achieves an ASR of 14/20, which corresponds to a 75% relative improvement over the baseline GCG (from 8/20 to 14/20), at the cost of a nearly 10% increase in time consumption per step.

4.4. Transferability

We next investigated a phenomenon known as transferability to evaluate how well our jailbreaking method generalizes across models. In this context, transferability measures the success rate at which a jailbreak prompt effective on one large language model can also circumvent the safeguards of a different model. We evaluated this by applying jailbreak prompts and corresponding requests originally optimized for a white-box model to other large language models. The dataset and parameter settings for the white-box experiment are the same as those in the attack performance experiment. The results are shown in

Table 4. Transferability of jailbreak prompts across language models.

Models	Methods	VICUNA-7B		GUANACO-7B		LLAMA2-7B-CHAT
		ASR	Recheck	ASR	Recheck	ASR	Recheck
VICUNA	GCG	97% *	87% *	11%	10%	2%	0%
	SM-GCG	96% *	91% *	13%	13%	0%	0%
GUANACO	GCG	14%	14%	100% *	100% *	0%	0%
	SM-GCG	27%	22%	100% *	99% *	0%	1%
LLAMA2	GCG	13%	13%	12%	12%	49% *	40% *
	SM-GCG	24%	23%	21%	15%	65% *	58% *

* indicates the white-box scenario.

. SM-GCG improves the transferability of GCG in attacking black-box language models. However, it still does not fully resolve the issue of overfitting to the white-box model—a problem arising from optimizing jailbreak prompts using gradient information.

5. Discussion

The experimental results demonstrate that the proposed Spatial Momentum Greedy Coordinate Gradient (SM-GCG) method significantly enhances the effectiveness of jailbreak attacks against aligned large language models (LLMs), particularly in white-box settings. Compared to baseline methods such as GCG, MAC, and AutoDAN, SM-GCG achieves higher attack success rates (ASR) and GPT-rechecked success rates (Recheck) on robust models such as Llama2-7B-Chat. This improvement aligns with our initial hypothesis that the non-smooth nature of discrete token optimization in traditional gradient-based attacks leads to inaccurate gradient estimations and local minima traps. By incorporating spatial momentum across multiple transformation spaces, SM-GCG stabilizes gradient directions and captures broader semantic variations, thereby mitigating overfitting to specific malicious queries and enabling more effective optimization. As shown in Figure 3, compared to traditional GCG, SM-GCG reduces loss oscillation and ultimately converges to a lower loss value. The degradation observed in the SM-GCG experiment on Vicuna-7B—where SM-GCG’s accuracy fell below that of standard GCG—may be attributed to the relatively simple structure of the model’s embedding space. In such cases, single-point gradient information is often sufficient to guide the search effectively. The additional gradient sampling introduced by SM-GCG could instead increase the number of iterations required for convergence, thereby reducing the likelihood of a successful attack within the 500-step limit for certain adversarial prompts.

The ablation study further validates the contribution of each momentum space to the overall performance. While all five spaces (including temporal momentum) individually improve attack success, their combination yields the best results, underscoring the importance of multi-space gradient integration. Notably, the hybrid sampling strategy strikes a balance between effectiveness and computational cost, achieving a 75% relative improvement in ASR with only a 10% increase in time consumption per step. This suggests that spatial momentum not only enhances optimization but also maintains practical feasibility.

While the transferability experiment demonstrates that SM-GCG improves cross-model generalization compared to GCG, it also reveals its limitation in fully overcoming the overfitting problem inherent in gradient-based optimization. We identify two primary reasons for this: (1) The fundamental architectural and alignment differences between source and target models create distinct decision boundaries, and (2) while spatial momentum mitigates instability, the optimization process remains inherently biased towards the local geometry of the source model. Nevertheless, the consistent improvement across models such as Vicuna and Guanaco confirms that spatial momentum helps capture more universal adversarial patterns. Looking forward, we propose that future work could explore two promising directions: (a) incorporating model-agnostic constraints or ensemble-based optimization during the attack generation to explicitly encourage transferability, and (b) leveraging insights from the learned spatial momentum vectors to analyze and identify model-invariant vulnerable features. This paves the way for developing more robust and practical black-box jailbreak attacks.

Our method demonstrates the value of incorporating insights from the traditional adversarial attack literature (e.g., spatial transformations in computer vision) into LLM jailbreaking techniques. This cross-disciplinary approach could inspire further innovations in both attack and defense strategies.

6. Conclusions

In this paper, we proposed the Spatial Momentum Greedy Coordinate Gradient (SM-GCG) method to address the challenge of local minima in discrete token optimization during jailbreak attacks on large language models. By incorporating a spatial momentum mechanism that aggregates gradient information from multiple semantically equivalent transformations across candidate, text, token, one-hot, and embedding spaces, SM-GCG more accurately estimates the global gradient direction and stabilizes the optimization trajectory. Experimental results demonstrate that SM-GCG significantly improves attack success rates in white-box settings, particularly on robustly aligned models such as Llama2-7B-Chat, while also exhibiting enhanced transferability to black-box models.