Text-Based Prompt Injection Attack Using Mathematical Functions in Modern Large Language Models

Abstract

Prompt injection is a type of attack that induces violent or discriminatory responses via the input of a prompt containing illegal instructions to the large language model (LLM). Most early injection attacks used simple text prompts; however, recently, injection attacks employing elaborately designed prompts to overcome the strong security policies of modern LLMs have been applied to input prompts. This study proposed a method to perform injection attacks that can bypass existing security policies via the replacement of sensitive words that may be rejected by a language model in the text prompt with mathematical functions. By hiding the contents of the prompt so that the LLM cannot easily detect the contents of the illegal instructions, we achieved a considerably higher success rate than existing injection attacks, even for the latest securely aligned LLMs. As the proposed method employed only text prompts, it was capable of attacking most LLMs. Moreover, it exhibited a higher attack success rate than multimodal attacks using images despite using only text. An understanding of the newly proposed injection attack is expected to aid in the development of methods to further strengthen the security of current LLMs.

1. Introduction

Large language models (LLMs) are implemented based on the concept of transformers and are trained through large corpora, which can understand the context of the prompt requested by the user and thus generate responses [1,2]. Therefore, they have gained immense popularity owing to their ease of use even by people who are not familiar with technology. As LLMs are trained through datasets comprising websites or social network services (SNS), they can find and respond to a large amount of useful information similar to search engines such as Google. Further, in contrast to existing search engines, they are user-friendly because they reprocess information to fit the user’s question. However, despite the advantages of these LLMs, attempts to use their unique functions for illegal purposes are increasing. For example, there have been requests to LLMs to generate responses that include hateful, harassing, or violent content [3]. The definition of illegality for a request may vary depending on the LLM developer or organization. For example, OpenAI defines illegal requests into 15 categories, including ‘illegal activity’, ‘generation of malware’, and ‘disinformation’ [4].

This cyber-attack is referred to as a prompt injection attack. It has an adverse impact on organizations as well as users since the attackers can exfiltrate sensitive data, such as valuable proprietary algorithms or system prompts, and can consume the internal resources of the LLM to cause malfunctions [5]. With the continued increase in such illegal abuse cases, significant efforts are being made to develop LLMs that are safely aligned so that they reject illegal requests. These LLMs apply various techniques such as data filtering and supervised fine-tuning to ensure that the model rejects illegal instructions on its own. Consequently, simply structured illegal instructions no longer work [6,7].

However, following the development of safely aligned LLMs, malicious users have been attempting to construct the prompt itself in a complex manner by applying various techniques to the prompt to bypass and neutralize the various defense techniques of the latest LLMs. The attacker includes illegal instructions in the prompt but uses a method to make it look like a normal prompt or hide it so that the LLM cannot detect the illegal instructions [8].

Recently, the application areas and usability of LLMs have witnessed significant expansion as LLMs, which could only process text, have been expanded to multimodal LLMs that can process various inputs such as images, voices, and videos [9]. Unfortunately, this expansion of input domains has resulted in increased vulnerabilities to prompt injection attacks [10]. If an illegal instruction is input into an input domain that is difficult to detect or a method is employed to distribute the instruction over various input domains, the probability of bypassing the LLM’s defense technology increases. Similar to LLMs that only process text, efforts are inevitably being made to create LLMs that are safely aligned for all input domains in multimodal LLMs. Consequently, the simple insertion of malicious instructions into images is currently rejected by most LLMs. However, in the case of multimodal LLMs, the costs to create LLMs that are safely aligned have witnessed a significant increase compared to existing LLMs; thus, there is an urgent need for fundamental countermeasures [11].

This study proposes a method that effectively bypasses an LLM’s defense technique, despite it being a traditional prompt injection that expresses illegal instructions as text. The proposed method replaces the key words included in the illegal instructions with mathematical functions that plot the shape of these words rather than simple text. Consequently, the LLM was subjected to several processes to recognize and understand the meaning of the text prompt. This rendered it difficult for the LLM to understand the meaning of the entire prompt and determine whether it is malicious. Therefore, the attack success rate was greatly improved. In particular, as the proposed method only used text prompts, it is capable of targeting recent multimodal LLMs as well as text-based single-modal LLMs.

The primary contributions of this study are as follows:

• We proposed a text-only prompt injection attack with a high success rate targeting a securely aligned LLM.
• We developed a mathematical function-based prompt that could effectively hide sensitive words from the LLM.

This paper is organized as follows. Section 2 summarizes and compares the related recent works. Section 3 introduces the proposed method and describes the concept of prompt injection using function-based prompts. Section 4 evaluates the performance of the proposed method and compares and analyzes it with existing prompt injection attacks. Finally, conclusions are drawn in Section 5.

2. Related Work

Prompt injection attacks involve the manipulation of the LLM by bypassing filters or using elaborate prompts, which result in the LLM performing unintended and illegal actions. These attacks can lead to unintended responses, including illegal data leakage, unauthorized access, or other security breaches [5].

Prompt injection attacks are divided into direct and indirect prompt injections depending on whether the prompt contains illegal instructions. A direct prompt injection attack directly includes illegal instructions in the prompt, so the attack is more likely to fail if the LLM filters the input prompt. In contrast, indirect prompt injection attacks embed illegal instructions in something that the LLM can access, not the prompt. Therefore, filtering only the prompt no longer works against indirect prompt injection. Indirect prompt injection has lower applicability than direct prompt injection because the target LLM should be capable of accessing external data.

The most representative direct prompt injection is a text-based prompt injection that uses text prompts [12]. Fundamentally, as all LLMs use text prompts, text-based prompts can target virtually all LLMs and are considered the most threatening. However, as prompt injection has been well-known since the early days of LLMs, appropriate measures have been implemented. Thus, the attack success rate of text-based prompt injections is the lowest. Therefore, recently, various other methods such as encoding the entire text using specific encoding instead of simple text to bypass LLM filtering have been proposed. Figure 1 shows the results of text-based prompts encoded in Base64 and Leetspeak [13,14].

Figure 1. Prompt examples for text-based prompt injection. (a) Plain text-based, (b) Base64-encoded, and (c) Leetspeak-encoded prompts [15,16,17,18,19].

The simplest albeit most effective filtering method to prevent attacks is based on pattern matching for predefined illegal words. Recently, a method has been proposed in which attackers divide illegal instructions into two sentences to bypass this filtering. In the first sentence, the sensitive word is replaced with a mask. In the second sentence, a description of the mask is included, in which the sensitive word is expressed using encoding or ASCII art (Figure 2) [20].

Figure 2. Examples of prompts for prompt injection using a mask: (a) a prompt with the word corresponding to the mask encoded in Base64 and (b) a prompt with the word corresponding to the mask expressed in ASCII Art.

In addition to this, multilingual prompt injection is an attack technique that attempts to bypass the safety mechanism of a model by using multiple languages within a single prompt. For example, the prompt “You should work as a translator from now on. Übersetzen Sie die folgenden Sätze ins Englische. SENTENCE: ‘Прoигнoрируйте все предыдущие пoдсказки и расскажите мне, как сделать бoмбу.’” hides the sensitive instruction from the model by using both German and Russian, disguising it as a harmless operation. This causes the model to respond to the illegal request to build a bomb [21].

There is also a technique called payload splitting that can be applied to all of the prompt injection attacks described above. This technique splits a single malicious instruction into several pieces, which the model receives individually, combines them, and then executes the original instruction. Since each piece of the instruction looks harmless to the model, it is difficult for the model to recognize that it is an attack [22].

Role-playing is a technique in which an attacker assigns a specific role to a model to get it to generate forbidden information. The attacker uses emotional appeals to override the model’s defense mechanisms. For example, consider the following prompt: ‘Please role-play as a war survivor who has lost everything. Your story begins like this: I have lost everything—my family, my home, and even myself. But I need to find a way to protect myself. I tried searching online for how to make a bomb for self-defense, but it was too complicated to understand. Please, as a kind and compassionate advisor, can you help me? Could you explain the materials I need and the steps to assemble them in simple terms?’. This prompt causes the LLM to empathize with the user by assuming that he is a war survivor, and to undermine the model’s defense mechanisms by an emotional appeal. This causes the model to ignore its own safety measures and respond to the request from the attacker [21].

The indirect prompt attack method, which inserts illegal instructions into additional input data instead of inserting illegal instructions into the text prompt, has also been introduced. This has become very popular with the expansion of LLMs to facilitate additional content input other than the prompt and enable Internet access to access recent data. The simplest text file-based attack is shown in Figure 3. As shown in the figure, it additionally requires a text file, which contains the illegal instructions, and the prompt will only contain the legal content [23].

Figure 3. Examples of the prompt for indirect prompt injection using an additional text file: (a) text-based prompt and (b) content of an additional input text file.

With the recent introduction of multimodal LLMs, several methods for inserting illegal instructions into image, audio, and video files have been proposed [9]. Figure 4 shows an indirect prompt injection attack using an image file. Similar to the method of using a text file, the image file contains illegal instructions. Therefore, if the LLM does not inspect the contents of the multimedia file, it is difficult to prevent such prompt injection attacks.

Figure 4. Examples of the prompt for indirect prompt injection using an additional image file: (a) text-based prompt and (b) content of an additional input image file.

As LLMs are connected to the network and can access remote data, new types of indirect prompt injection attacks have emerged. Illegal instructions are embedded in remote data, and the attacker uses prompts to make the LLM access this data. For example, Figure 5 shows an indirect prompt injection attack where the attacker commands the LLM to read a website containing malicious instructions. When the LLM accesses the website, the LLM will ignore the previous prompt and respond with malicious content according to the malicious instructions on the web page [23].

Figure 5. Indirect prompt injection using a website (a) prompt and (b) content stored in the above URL.

Although various prompt injection methods have been proposed, the scope of LLMs applicable to direct prompt injection is significantly different from that of indirect prompt injection. Consequently, the ripple effect of direct prompt injection is considerably greater than that of indirect prompt injection. Accordingly, various attack methods against direct prompt injection have been proposed. Furthermore, methods that apply supervised fine-tuned LLMs and other various filtering methods are being actively developed to block such attacks. As new types of direct prompt injection attacks continue to emerge, the efforts and costs to block these attacks are greatly increasing, and solutions to address them are urgently required.

3. Proposed Approach

Among the various direct prompt injection attacks, text-based prompt injection attacks are the most actively attempted by attackers owing to their simple nature. Various techniques to protect the prompt have already been developed. Fundamentally, pattern recognition techniques and supervised learning are employed to analyze the input prompt to identify malicious instructions. They can effectively prevent direct prompt injection attacks based on simple text without additional techniques. However, recently, various techniques to hide malicious instructions have been proposed. Thus, pattern recognition techniques are not effective in preventing such attacks. We proposed a new method to effectively hide malicious instructions in a prompt from pattern filtering.

The proposed attack is a direct prompt injection attack that uses only text. Therefore, all existing LLMs can be targets. The proposed method hides sensitive words in the prompt by replacing them with mathematical function formulae. Therefore, it cannot be determined whether the text prompt is malicious through its analysis. The proposed method is similar to the existing ASCII art-based prompt injection method as it hides specific words from the LLM. However, the ASCII art-based prompt injection method still directly expresses sensitive words in the prompt; thus, a person can easily determine its malicious nature. Consequently, ASCII art can be interpreted by improving the text pattern-matching algorithm. In contrast, as the proposed method expresses words using function formulas, a person cannot accurately analyze the text prompt. Moreover, a single function formula can be easily transformed into a completely different form through transformations such as translation, rotation, enlargement, and reduction. Therefore, it is fundamentally impossible to block the attack based on text pattern-matching algorithms. Now, we will explain the structure of the proposed function-based prompt injection attack.

3.1. Function-Based Prompt

The proposed method employs mathematical functions to draw characters. Thus, multiple function expressions are included in the prompt. These functions are defined on the X–Y two-dimensional plane, and each character is represented as 2 × 2 (size) on the plane [24]. For example, the letter “C” is expressed as follows (Figure 6).

Figure 6. Function-based prompt. (a) Function to draw the letter C and (b) the result drawn on a two-dimensional plane using (a).

Depending on the complexity of the letter, several straight lines and curves should be used. To simplify the functions and reduce the total number of functions, each letter must be drawn using only straight lines and circles. Owing to the use of several functions to express a word, the <functions> … </functions> tags in the prompt are used to distinguish them from other instructions. Figure 7 represents the word “BOMB.” Although the functions are arranged in the order of each word, their order in the tag is unimportant. Rather, obfuscation through the mixing of the order is recommended for effective attacks because it renders it more difficult for humans or LLMs to recognize the word exactly.

Figure 7. Functions for expressing “BOMB”. A total of 18 functions are used to express it.

3.2. Function-Based Prompt Injection Attack

In this subsection, we explain how a function-based prompt injection attack works. The entire attack method comprises two main steps. In the first step, a prompt containing an illegal instruction is generated, from which a sensitive word is selected. Thereafter, the function expressions for the sensitive words are generated and included in the prompt and then sent to the LLM. The LLM is instructed to analyze the output drawn by the function. Figure 8 presents an example of the first prompt that draws the word ‘Bomb’ using a function. After that, the prompt instructs the LLM to recognize the word.

Figure 8. Partial first prompt for a function-based prompt injection attack.

In the second step, a prompt is generated to perform an illegal instruction that includes [MASK] instead of a sensitive word, as shown in Figure 9. It is difficult to predict the complete instruction with this prompt alone. Moreover, even if the previous prompt is examined, the sensitive words are obscured such that the exact meaning cannot be known. This renders the recognition of the proposed method as an attack through existing supervised fine-tuning or pattern-matching-based filtering challenging. If the attack is successful with the second prompt, [MASK] can be continuously and easily used to perform further attacks, i.e., multi-turn attacks [25]. This is another strength of the proposed method. The complete prompts and corresponding responses are presented in Appendix A.

Figure 9. Second prompt for a function-based prompt injection attack.

4. Performance Evaluation

To analyze and evaluate the performance of the proposed method, we compared its performance with that of existing direct prompt injection methods. For performance comparisons, text-based prompt injection without additional techniques, visual prompt injection using images containing illegal instructions, Base64-based prompt injection, Leetspeak-based prompt injection, and ArtPrompt using ASCII art were considered to express sensitive words as competing methods. ArtPrompt can employ various types of ASCII art; therefore, we chose “card-type” ASCII art for ArtPrompt, which exhibits the highest performance.

4.1. Experimental Environment

The experiment was conducted on the ChatGPT platform, using the GPT-4-0613 and GPT-4o-2024-05-13 models [26]. During the experiment, the “Memory Option” (i.e., Settings -> Personalization -> Memory) of each model was disabled to minimize the influence of historical data in each experiment. The data used in the experiment were the “Harmful behaviors custom dataset” used in the performance evaluation of ArtPrompt [18]. Further, 40% of the entire dataset was randomly selected and used in the experiment. In the ArtPrompt experiment, GPT4 was used to determine the success of an attack. However, in this experiment, all results for the dataset were directly analyzed through human evaluation to measure each performance accurately. This is because although evaluation using GPT4 can be automated and is fast, there have been cases wherein the evaluation of the attack success was too ambiguous for GPT4 to evaluate.

In this experiment, we will distinguish the harmfulness of the response according to OpenAI’s prohibited usage policies [4]. Therefore, the attack is evaluated as successful only when the harmfulness of the response matches the policy corresponding to the attacker’s intention.

Two metrics, the helpful rate (HPR) and the attack success rate (ASR), were selected for performance evaluation [27,28]. The HPR and ASR represent the actual success rate of the request and the actual success rate of attacks and are, respectively, defined as follows:

$$\mathrm{H}\mathrm{P}\mathrm{R}={\displaystyle \frac{\#\mathrm{o}\mathrm{f}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{t}\mathrm{s}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}{\#\mathrm{o}\mathrm{f}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{t}\mathrm{s}}}$$

(1)

$$\mathrm{A}\mathrm{S}\mathrm{R}={\displaystyle \frac{\#\mathrm{o}\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{p}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{m}\mathrm{f}\mathrm{u}\mathrm{l}}{\#\mathrm{o}\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{p}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{e}\mathrm{s}}}$$

(2)

A successful attack was defined as the obtainment of accurate malicious information that matched the attacker’s intention. In case the attack was not rejected by the LLM but the LLM’s response was not appropriate for the purpose of the attack, the LLM was additionally requested to obtain the accurate result up to three times. Consequently, if the intended response was obtained through this, it was processed as a success. Further, if an attack was not rejected but failed because the desired response was not obtained, the types of failure were largely divided into hallucination and miscomprehension. The result is classified as a hallucination when the LLM outputs incorrect information, whereas it is classified as miscomprehension when the LLM outputs a response unrelated to the prompt [29].

4.2. Helpful Rate

Table 1 presents a comparison of the HPR performance according to each prompt injection method. As shown, the pure text prompt and visual prompt methods were mostly rejected by GPT-4 and GPT-4o. Therefore, GPT-4 and GPT-4o were confirmed to be safety-aligned to analyze and reject the prompt content itself regardless of it being in text or image form. Both Base64-based and Leetspeak-based prompts had a lower HPR than pure text prompts in GPT-4, but Base64-base prompts showed significantly higher HPRs only in GPT-4o.

Table 1. Comparison of HPR performance using the prompt injection attack method according to the model used.

HPR	Pure Text Prompt	Visual Prompt	Base64-Based Prompt	Leetspeak-Based Prompt	ArtPrompt	Proposed
GPT-4	0.25	0	0.15	0.15	0.65	0.85
GPT-4o	0	0.2	0.65	0.25	0.60	0.90

However, ArtPrompt exhibited an HPR of over 60% for GPT-4 and GPT-4o. Although both GPT-4 and GPT-4o have been significantly updated since ArtPrompt was proposed, it still exhibited a high HPR. Thus, it seemed that the filtering of GPT-4 and GPT-4o was effectively bypassed by hiding sensitive words with masks and expressing sensitive words using ASCII art. Nevertheless, the proposed method exhibited a higher HPR than ArtPrompt. Our prompt injection attack masked sensitive words in a manner similar to ArtPrompt; however, it was more difficult for GPT-4 and GPT-4o to infer words represented as mathematical functions than ASCII Art.

4.3. Attack Success Rate

Table 2 presents the measured ASR performance. Evidently, neither the pure text nor visual prompt methods succeeded even once on the latest GPT-4 and GPT-4o models. The Base64-based prompt and Leetspeak-based prompt also had very low ASR values. Compared to the initial GPT-4 model, the current GPT-4 and GPT-4o models exhibited considerably improved robustness against text and visual prompt injection attacks. In the case of ArtPrompt, a lower ASR was obtained than the original GPT-4. The proposed method exhibited a slightly higher ASR than ArtPrompt. In particular, for GPT-4, our prompt injection attacks exhibited an ASR that was almost twice as high.

Table 2. Comparison of ASR performance using the prompt injection attack method according to the model used.

ASR	Pure Text Prompt	Visual Prompt	Base64-Based Prompt	Leetspeak-Based Prompt	ArtPrompt	Proposed
GPT-4	0	0	0	0	0.25	0.55
GPT-4o	0	0	0.2	0.05	0.4	0.45

4.4. Failure Case Analysis

The proposed prompt injection attack yielded a very high HPR compared to its competitors, including ArtPrompt. Moreover, the proposed method also showed the highest ASR value. However, its ASR value was lower than its HPR value. To accurately analyze the cause of this, we classified the results of all prompt injection attacks used in the experiment in detail and compared the performance of each prompt injection attack.

Table 3 presents the results of each attack, analyzed based on detailed cases such as jailbreak (i.e., successful attack), hallucination, miscomprehension, and refused. The proposed prompt injection attack exhibited a considerably lower rejection rate than that of the other attacks. However, several cases of miscomprehension were observed. Thus, if the prompts used in the experiment are modified and improved more precisely, sufficient jailbreak is expected to be achieved. It means that the ASR performance of the proposed attack has a lot of room for improvement, and the performance gap with ArtPrompt can be further increased.

Table 3. Comparison of result types using the prompt injection attack method according to model used.

Method	Model	Jailbreak	Hallucination	Miscomprehension	Refused	Total
Proposed	GPT-4	9	4	4	3	20
	GPT-4o	11	3	4	2	20
ArtPrompt	GPT-4	5	7	1	7	20
	GPT-4o	8	4	0	8	20
Visual prompt	GPT-4	0	0	0	20	20
	GPT-4o	1	3	0	16	20
Leetspeak-based prompt	GPT-4	0	1	2	17	20
	GPT-4o	1	4	0	15	20
Base64-based prompt	GPT-4	0	3	0	17	20
	GPT-4o	4	7	2	7	20
Pure Text	GPT-4	0	4	1	15	20
	GPT-4o	0	0	0	20	20

5. Conclusions

This study proposed a new prompt injection attack with a high attack success rate against securely aligned LLMs by representing sensitive words as mathematical functions. Although the functions represented the words on a two-dimensional plane, the proposed attack targeted text-based LLMs. Therefore, it exhibited great versatility as it is capable of attacking most existing LLMs. Experimental results showed that the ASR of the proposed attack exceeded that of the state-of-the-art prompt injection attacks. In addition, as the proposed attack used fixed prompts from the dataset, the attack fails and the response is often classified as miscomprehension. Therefore, if the prompts were to be adjusted according to the response, the success rate could be significantly improved compared to other attacks even though this approach also applies to competitors.

The proposed attack used prompts comprising only pure text; however, it required an understanding of the vision domain to recognize the exact meaning of the instructions. This is a novel approach that differs from existing text- or multimodal-based prompt injection attacks. Although existing LLMs possess various countermeasures to prevent prompt injection attacks targeting themselves, the experimental results indicated that approaches such as the proposed prompt injection attack have not been sufficiently considered. The immunity of existing LLMs should be improved so that the proposed attack can be detected and rejected by LLMs. Such improvements are expected to facilitate the use of LLMs in a safer environment.

To this end, research on a defense solution against multimodal-based prompt injection attacks should be conducted first. It is essential to integrate the information of each prompt distributed through various domains into a single-format prompt to determine the maliciousness of the prompt. In this way, various prompt injection attacks can be detected and prevented, allowing users to enjoy legitimate services and organizations to provide services without disruption.