Prompt Optimization with Two Gradients for Classification in Large Language Models

Abstract

Large language models (LLMs) generally perform well in common tasks, yet are often susceptible to errors in sophisticated natural language processing (NLP) on classification applications. Prompt engineering has emerged as a strategy to enhance their performance. Despite the effort required for manual prompt optimization, recent advancements highlight the need for automation to reduce human involvement. We introduced the PO2G (prompt optimization with two gradients) framework to improve the efficiency of optimizing prompts for classification tasks. PO2G demonstrates improvement in efficiency, reaching almost 89% accuracy after just three iterations, whereas ProTeGi requires six iterations to achieve a comparable level. We evaluated PO2G and ProTeGi on a benchmark of nine NLP tasks, three tasks from the original ProTeGi study, and six non-domain-specific tasks. We also evaluated both frameworks on seven legal-domain classification tasks. These results provide broader insights into the efficiency and effectiveness of prompt optimization frameworks for classification across diverse NLP scenarios.

1. Introduction

Large language models (LLMs) are widely used in natural language processing tasks and have demonstrated strong performance in many use cases [1]. However, they often struggle with sophisticated problems [2]. Prompt engineering, the craft of designing targeted prompts to guide model outputs, has emerged as a strategy for improving LLM performance [3]. In this research, we focus on using LLMs to annotate legal documents. LLMs have shown promising results in the legal domain, GPT-4 successfully passed the bar exam [4], and other studies demonstrated its potential as a legal assistant, achieving roughly a C+ grade on law school exams [5,6]. However, LLM performance remains inconsistent on sophisticated problem tasks like legal text annotation, indicating the need for better techniques.

Two widely used prompt-engineering techniques are chain-of-thought prompting and few-shot prompting. Chain-of-thought (CoT) prompting involves guiding the model to produce step-by-step reasoning before giving a final answer, effectively simulating human-like reasoning [3]. Few-shot prompting improves performance by providing a few example input-output pairs within the prompt to help the model infer the task format [7]. While both methods can improve accuracy, they require substantial human expertise to craft effective prompts and often involve collaboration with domain experts. These prompts are also computationally expensive at inference time due to the high token counts in the model’s output reasoning. Therefore, there is a need for automated prompt optimization frameworks that can generate concise, direct prompts with better accuracy and consistency, without extensive human effort.

1.1. Literature Review

1.1.1. Survey Paper

Prompt engineering has been the subject of several comprehensive surveys.

Table 1. Survey papers on prompting and automatic prompt engineering.

Reference	Method	Limitation
White et al. [8]	Prompt Pattern Catalog (survey)	—
Liu et al. [9]	Pretrain–prompt–predict survey	—
Li et al. [10]	Optimization-based prompting survey	—

highlights a few of these works [8,9,10]. Collectively, these surveys review the paradigm shift from the traditional pretrain–fine-tune approach to a pretrain–prompt–predict paradigm, and catalog a wide range of prompt design techniques. Following this literature, we can broadly classify prompt optimization methods into two families based on the behavior of the optimized prompt: (i) analysis-driven prompts, which encourage the LLM to produce reasoning or explanations, and (ii) direct-label prompts, which aim for the LLM to output a final answer or class label directly without reasoning.

1.1.2. Foundation Inspiration

Table 2. Foundational prompt-engineering methods that inspired our work. We group them as analysis-driven (reasoning) vs. direct-label (produces labels without explicit reasoning).

Reference	Method	Limitation
Analysis-driven methods
Wei et al. [3]	Chain-of-Thought	Requires curated prompt; higher inference cost due to longer reasoning chains/tokens
Zhang et al. [11]	Auto-CoT	LLM-generated prompt; inconsistent on multiple run;higher inference cost due to longer reasoning chains/tokens
Sun et al. [12]	AutoHint	Actor–critic loop over a single evolving prompt; hint may encourage prompt to analyze and reason
Direct-label methods
Pryzant et al. [13]	ProTeGi	Bandit/minibatch scoring; no explicit separation of FP vs. FN errors
Zhou et al. [14]	CLAPS	Evolutionary selection without explicit error feedback
Dong et al. [15]	PACE	Optimizes one candidate prompt per step, not tailored to classification; no explicit separation of FP vs. FN errors
Yuksekgonul et al. [16]	TextGrad	complex framework; better fit to complex reasoning/ than classification

lists foundational prompt-engineering methods that inspired our work, grouped by whether they use an analysis-driven approach or a direct-label approach. In the analysis-driven category, the chain-of-thought (CoT) method uses manually crafted multi-step reasoning in the LLM’s output [3]. This approach can improve performance but requires carefully curated prompts and incurs a higher inference cost due to the longer reasoning tokens. Auto-CoT [11] attempts to automate this process by using the LLM itself to generate reasoning chains. However, its outputs can be inconsistent across multiple runs and still require reasoning output. AutoHint [12] iteratively adds hint phrases to a single evolving prompt. These hint-based prompts may encourage the model to analyze and reason through the problem, but the method still focuses on expanding the reasoning on inference.

In the direct-label category, the goal is to produce the correct label or decision with minimal reasoning in the inference. ProTeGi [13] applies gradient-inspired textual edits to prompts, using a bandit or minibatch evaluation strategy incrementally. This method demonstrated that LLM prompts can be optimized by treating prompt edits as gradient steps; however, it does not explicitly distinguish between different types of errors (false positives and false negatives) when scoring edits. CLAPS employs an evolutionary algorithm: it maintains a pool of candidate prompts and iteratively selects, mutates, and recombines them, guided by their performance [14]. CLAPS can produce robust prompts for document annotation tasks. Still, it lacks a mechanism to use specific error feedback. PACE uses an actor–critic model to refine a single prompt in an iterative loop [15]. PACE can handle direct labeling tasks, but most demonstrations of PACE have been on non-classification tasks, and it optimizes one prompt at a time without explicitly handling different error types. TextGrad generalizes the idea of gradient-guided prompt editing by formulating prompt optimization as an end-to-end differentiable problem [16]. TextGrad is a complex framework that can handle complex reasoning prompts. Importantly, none of these automatic prompt optimization methods explicitly separates false positive (FP) and false negative (FN) errors when guiding prompt refinements. Our work builds on the direct-label paradigm with a focus on efficiency and introduces a novel way to incorporate error-specific feedback into prompt optimization.

1.1.3. Related Work

Beyond the foundational methods, several recent works have proposed new frameworks for prompt optimization, as summarized in

Table 3. Recent studies related to our framework.

Reference	Method	Limitation/Notes vs. Our Framework
Yang et al. [17]	AMPO	Expands and keeps a pool of candidates; evaluated mainly on instruction-following/reasoning tasks.
Wu et al. [18]	STRAGO	Generates guided edits with additional “thinking” tokens; analysis-oriented, increasing token cost.
Choi [19]	APO-CF	Uses aggregate confusion-matrix signals (not per-instance FP/FN); demonstrated mainly for relevance evaluation; limited search depth per round.
Kumar et al. [20]	SCULPT	Designed a systematic way to edit long prompts, may miss on the other structure not defined.
Das et al. [21]	GREATER	Requires token-level logits/probabilities unavailable in closed LLM.
Jain and Chowdhary [22]	LPO	Token-level/local edits can miss larger structural changes.
Mayilvaghanan et al. [23]	PROPEL	Agent-loop with cost overhead; evaluated mainly on small/medium LLMs.
Wu et al. [24]	AutoMedPrompt	Medical QA-centric; single-candidate refinement.

. AMPO [17] expands and maintains a diverse pool of prompt candidates (a multi-branch optimization strategy), which can improve robustness but also increases the number of LLM calls needed per iteration. STRAGO [18] introduces guided edits by adding reasoning tokens to prompts; this approach steers the LLM’s reasoning process, but similarly to CoT, it results in longer prompts and higher inference costs focused on analytic reasoning tasks.

Another line of work tries to leverage performance feedback more directly. APO-CF (automatic prompt optimization with confusion matrix feedback) [19] uses the model’s confusion matrix (on a validation set) to guide prompt edits. This method provides a form of error feedback, but it uses aggregated signals (like overall false positive/negative counts) rather than pinpointing individual instances and their mistakes. Other approaches focus on the structure of the prompt itself: SCULPT [20] systematically tunes sections of a long prompt (assuming the prompt has a predefined structure with segments), which is useful for very structured prompts but may miss more free-form changes. GREATER [21] proposes using gradients over the model’s reasoning process (requiring access to token-level probabilities) to refine prompts, an approach that is not feasible with closed-source LLMs where internal logits are inaccessible.

Some methods aim for fine-grained edits or low-resource scenarios. LPO (local prompt optimization) [22] looks at token-level replacements in a prompt based on local utility, which can improve a prompt iteratively; however, focusing only on local edits might overlook the need for larger structural or semantic changes in the prompt. PROPEL (prompt optimization with expert priors for LLMs) [23] uses an agent-based loop that incorporates expert knowledge or heuristic rules during prompt refinement; it has mainly been tested on small and medium-sized LLMs, and the additional agent overhead can increase optimization time. AutoMedPrompt [24] adapts textual gradient methods specifically for medical question-answering, refining a single prompt candidate through multiple iterations. This domain-specific approach shows the versatility of prompt optimization techniques, though it was demonstrated on specialized medical tasks. Notably, none of the above methods explicitly leverage separate FP vs. FN as distinctive signals in their optimization loops, and, to our knowledge, none have been applied to legal document annotation scenarios. The framework discussed in this section was recently published and was not available at the time PO2G was developed in Q3 of 2024.

1.2. Framework with Two Gradients

Inspired by the strengths and limitations of ProTeGi, CLAPS, AutoHint, and PACE, we propose the PO2G (prompt optimization with two gradients). The PO2G framework uses a gradient-descent-like iterative approach to refine prompts. The key innovation is that it categorizes the model’s misclassifications into two types: false positives (FP) and false negatives (FN), and treats each as a “distinct gradient” or direction for improvement. In each iteration, the framework gathers the examples from the LLM’s incorrect predictions and splits them into an FP set (data should be excluded) and an FN set (data should be included). This separation allows PO2G to make targeted adjustments to the prompt; the prompt can be edited to be more restrictive in response to FPs and more inclusive or instructive in response to FNs. This framework simulates human behaviour to update the prompt when it finds an incorrect prediction. Additionally, to avoid overwhelming the prompt with too much similar feedback, we employ a clustering step on the incorrect examples. By clustering the FP examples and FN examples separately, PO2G takes sample representative error cases from each cluster. The prompt is then updated with focused guidance addressing those representative cases, which reduces redundancy in the feedback and keeps the prompt efficient.

PO2G is a data-driven framework designed to optimize prompts more efficiently than ProTeGi. It takes advantage of modern LLMs’ larger context windows. GPT-3.5 Turbo output token is limited to 4096 tokens; newer models, such as GPT-4o-mini, can handle significantly longer prompts. This expanded context capacity allows LLM to include more detailed feedback and instructions in each iteration step. Figure 1 illustrates an example of prompt refinement with PO2G. A simple initial prompt (iteration 0) is gradually transformed into a more detailed and effective prompt by iteration 3, with additional instructions and clarifications incorporated at each step. In this way, PO2G iteratively builds and edits a prompt that captures the optimal instructions for the task, without requiring a human to write out long reasoning chains or numerous examples manually. PO2G frameworks support different LLMs at any stage. A more powerful LLM can be used for feedback and prompt generation, while a cheaper model handles scoring.

The ProTeGi framework was able to optimize prompts for domain-specific tasks. However, the author identified issues such as overfitting, limited task diversity, and inefficiency. PO2G (prompt optimization with two gradients) is an alternative framework to ProTeGi by addressing inefficiency with a different strategy to optimize the prompt. We evaluate prompt optimization across nine NLP tasks, with an additional seven Legal NLP tasks, thereby adding diversity to the NLP task set. To address overfitting, P02G processes the whole training dataset instead of mini-batching to mitigate local bias and ensure comprehensive learning. Utilizing clustering on all datasets helps identify the most influential prompts to improve efficiency.

1.3. Legal LLM

A major motivation for PO2G is its application to legal document analysis. In practice, legal professionals often need to extract obligations, requirements, or other information from lengthy documents. We integrate legal cases into our evaluation for behavioral analysis and compare our method with ProTeGi for tax and obligation document extraction. However, due to data limitations, we conduct the testing with publicly available data to compare the performance between PO2G and ProTeGi. The result in the legal domain will be shown in this research as an exploratory analysis.

Our proposed framework achieved a certain level of accuracy faster than ProTeGi in these legal NLP tasks. For instance, after three optimization iterations, the prompt generated by PO2G attained an accuracy comparable to what ProTeGi achieved after six iterations. This suggests that separating error types and categorizing feedback helps accelerate the learning of an effective prompt. For all tasks (both general and legal), we report accuracy as the primary evaluation metric to stay consistent with the metrics used by ProTeGi’s authors. However, we recognize that in the legal extraction tasks, the data are imbalanced, and the amount of data extracted are way less than the unrelated information. Therefore, in addition to accuracy, we also report precision, recall, and F1 score for the positive class (element that needs to be extracted) to better capture the model’s performance on the minority class. All metrics are computed by comparing the LLM’s predicted labels to the ground truth.

1.4. Research Contributions

In summary, this study advances prompt engineering research in several complementary ways:

Revisiting a baseline: We replicate the ProTeGi prompt optimization framework using a newer model, GPT-4o-mini. This updated baseline is important because larger-context models can incorporate longer feedback chains and generally exhibit stronger performance on complex tasks, providing a realistic comparison for our approach.

Broadening empirical coverage: We significantly expand the evaluation suite beyond the original ProTeGi paper’s four tasks. In our experiments, PO2G is tested on nine diverse public NLP tasks, and we further explore its application on seven legal document annotation tasks. This broad evaluation demonstrates the generalizability of our method and provides insight into its performance in a specialized domain (legal NLP) that was not covered in prior work.

Introducing the PO2G(P-O-Two-G) framework: We propose PO2G (prompt optimization with two gradients), a novel automatic prompt optimization framework. PO2G leverages a gradient-descent-inspired prompt editing strategy with support for large prompt contexts, treats false positives and false negatives as two distinct textual “gradients” for targeted feedback, and uses clustering of misclassified examples to reduce redundant information during optimization. This approach is particularly useful for classification tasks.

2. Methodology

This section introduces the PO2G Framework for prompt optimization (Figure 2). It begins with the hard prompt optimization process, followed by an explanation of the data sampling and clustering methods used to generate feedback for prompt refinement. Next, it discusses prompt expansion and selection strategies and concludes with a comparison between the PO2G Framework and ProTeGi.

2.1. Hard Prompt Tuning in Textual Gradient Descent

Hard Prompt Tuning, also known as discrete prompt optimization, is a process that refines prompts using human-readable words or tokens. Hard prompt tuning in textual gradient descent is an iterative process that adjusts a discrete prompt by navigating a defined textual space, guided by gradients derived from feedback on incorrect predictions. Section 2.6 illustrates this process. In our research, we extend this approach with PO2G prompt optimization, which simultaneously utilizes two feedback signals to improve the prompt.

2.2. Two Distinct Gradients

Algorithm 1 and Figure 3 outline the two distinct gradients on the PO2G framework. The process starts with an initial prompt $P_0$ and a labeled training dataset $D_{text,label}$. First, $LL{M}_1$ evaluates $P_0$ on $D_{text}$ to generate predictions. Incorrect predictions are identified as either false positives (FP) or false negatives (FN), forming two distinct loss signals.

Algorithm 1 Two gradients in the PO2G framework.

Require:  $P_0$: Initial prompt, $D_{text,label}$: Training dataset (text, label), $LL{M}_1$: LLM to classify,     $LL{M}_{2FP}$: LLM to generate FP feedback, $LL{M}_{2FN}$: LLM to generate FN feedback, $LL{M}_3$:     LLM to generate new prompt Ensure:  $P_{FP}^{\prime },P_{FN}^{\prime }$   1: Step 1: Evaluate Initial Prompt   2: $LL{M}_{1Prediction}\leftarrow LL{M}_1(P_0,D_{text})$   3: if$LL{M}_{1Prediction}\ne D_{label}$ then   4:        $D_{\left(FP\right)}$ : $LL{M}_{1Prediction}=1$ and $D_{label}=0$   5:        $D_{\left(FN\right)}$ : $LL{M}_{1Prediction}=0$ and $D_{label}=1$   6: end if   7: Step 2: Loss Signal Sampling   8: $Feedbac{k}_{FP}\leftarrow LL{M}_{2FP}(P_0,\mathrm{Sample}{D}_{\left(FP\right)})$   9: $Feedbac{k}_{FN}\leftarrow LL{M}_{2FN}(P_0,\mathrm{Sample}{D}_{\left(FN\right)})$ 10: Step 3: Generate a New Prompt 11: $P_{FP}^{\prime }\leftarrow LL{M}_3(P_0,Feedbac{k}_{FP})$ 12: $P_{FN}^{\prime }\leftarrow LL{M}_3(P_0,Feedbac{k}_{FN})$        return $P_{FP}^{\prime },P_{FN}^{\prime }$

In Step 2, feedback is generated by sampling from the $D_{\left(FP\right)}$ and $D_{\left(FN\right)}$ subsets using $LL{M}_{2FP}$ and $LL{M}_{2FN}$, respectively. This feedback guides the prompt refinement process via $LL{M}_3$, producing two new prompts: $P_{\mathrm{FP}}^{\prime }$ and $P_{FN}^{\prime }$. Section 2.6 illustrates an example of FP data and the corresponding feedback generation.

In Steps 2 and 3, the default sample size is set to five for each distinct gradient. The selection process may be conducted either randomly or through a clustering method, which will be detailed in the subsequent subsection. From the five samples, the prompt yielding the highest accuracy within each false positive (FP) and false negative (FN) subset is selected to represent the corresponding FP and FN gradients.

By leveraging both FP and FN data as two distinct gradient signals, our method provides targeted feedback to refine the prompt. Specifically, $LL{M}_{2FP}$ generates feedback instructing how to modify the prompt to exclude unnecessary data that were incorrectly included (FP), while $LL{M}_{2FN}$ produces feedback to incorporate data that should have been classified as positive but were missed (FN).

2.3. Loss Signal Selection: Cluster Sampling

The clustering process begins after evaluating $P_0$ using $LL{M}_1$, as illustrated in Algorithm 2 and Figure 4. Both the $D_{\left(FP\right)}$ and $D_{\left(FN\right)}$ datasets undergo clustering separately, with the default number of clusters set to 5. Initially, the $D_{\left(FP\right)}$ and $D_{\left(FN\right)}$ data are embedded into a vector space using the “embedding-3-small” model from OpenAI. This embedding method is chosen for its simplicity and cloud-based capabilities.

We use the K-means algorithm for clustering, as used on the clustering and pruning framework [14], due to its simplicity, speed, and effective centroid-based grouping. Notably, they also indicate that variations in the clustering method, and even the embedding technique, have minimal impact on the overall framework, reinforcing our choice of these methods. A sample closest to each cluster’s centroid is selected to represent that cluster. This approach reduces redundancy and ensures greater consistency compared to randomly selecting data points as loss signals.

Although the embedding and clustering processes require computational time, these steps are expected to reduce the overall computational load by minimizing the number of identical feedback and prompts that lead to more API calls to $LL{M}_1$, $LL{M}_{2FP}$, $LL{M}_{2FN}$, and $LL{M}_3$.

Typically, the number of prompts generated after the clustering process is twice the number of clusters. The most influential prompt across all clusters will be selected, which is the prompt with the highest accuracy. ($P_{FP}^{\prime },P_{FN}^{\prime }$) is based on the most influential prompt selected previously. These two new prompts are then compared and ranked against other prompts in the same iteration, which will be explained in the next subsection.

Algorithm 2 Clustering to find an influential sample.

Require: D: Input dataset (FN or FP data), num_clusters: Number of clusters to form,     embed: Text embedding function, K-Means: K-means clustering function Ensure:  representative_data: Set of data points representing each cluster   1: Step 1: Embed the input data   2:    $E\leftarrow \left\{\mathtt{embed}\right(x)\mid x\in D\}$ A   3: Step 2: Apply K-means clustering to the embedded data   4:    $\mathrm{clusters},\mathrm{centroids}\leftarrow \mathtt{KMeans}(E,\mathtt{num}\_\mathtt{clusters})$   5: Step 3: Select the closest point to each cluster’s centroid   6: for each cluster $k\in \{1,2,\dots ,\mathtt{num}\_\mathtt{clusters}\}$  do   7:        $x^{*}\leftarrow arg{min}_{x\in D_k}{\parallel \mathtt{embed}\left(x\right)-\mathrm{centroids}\left[k\right]\parallel }_2$   8:        Add $x^{*}$ to representative_data:   9: end for     return representative_data:

2.4. Prompt Expansion and Selection

Algorithm 3 and Figure 5 show how the binary tree structure emerges naturally due to the presence of two new prompts derived from the initial prompt. In each iteration, this process generates $P_{\mathrm{FP}}$ and $P_{\mathrm{FN}}$ from $P_0$. As the process repeats, the tree expands exponentially, with the number of nodes growing in powers of two: 1, 2, 4, 8, etc., making the algorithm highly computationally intensive and expensive in the long run.

Due to the high computational cost of the framework, a limit is placed on the expansion of nodes. The parameter max_prompt_expanded controls the maximum number of nodes that can be expanded (default = 2). As a result, if more than two new prompts are generated from a previous iteration, only max_prompt_expanded prompts will be expanded. This limitation is enforced by marking nodes as either active or inactive.

Only active nodes are expanded. An algorithm is used to identify the top-performing prompts at each level (iteration) of the tree. At each level, the algorithm selects the nodes with the highest accuracy, and the top-n prompts (where $n=\mathtt{max}\_\mathtt{prompt}\_\mathtt{expanded}$) are expanded. This selection strategy helps minimize computational effort by terminating low-performing nodes. The node termination process is illustrated in Section 2.6.

Algorithm 3 Prompt expansion and selection.

Require: $P_0$: Initial prompt, $D_{text,label}$: Training dataset (text, label), max_depth: Maximum   iteration of expansion, max_nodes_per_level: Maximum expanded nodes per level   (default = 2), $\mathcal{G}\left(\right)$: Two Gradients Function. Ensure: Optimized prompt $P^{*}$   1: Initialize: Active node and iteration   2: active_node_set $\leftarrow \{P_0\}$   3: $\mathrm{iteration}\leftarrow 0$   4: while  $iteration\le$  max_depth do   5:    for each $P_i\in$ active_node_set do     6:       Step 1: Generate new prompts using $\mathcal{G}\left(\right)$   7:       $P_{FP},P_{FN}=\mathcal{G}(P_i,D_{text,label})$     8:       Step 2: Collect the training accuracy of the newly generated nodes   9:                  $A_{FP}\leftarrow$ Train accuracy of $P_{FP}$ 10:                  $A_{FN}\leftarrow$ Train accuracy of $P_{FN}$ 11:                  Append $(P_{FP},A_{FP})$ and $(P_{FN},A_{FN})$ to accuracy_list[iteration] 12:    end for  13:    Step 3: Update active nodes for the next iteration 14:         Sort accuracy_list[iteration] by $A_i$ in descending order 15:         active_nodes $\leftarrow \mathtt{accuracy}\_{\mathtt{list}}_{\mathrm{iteration}}[0:\mathtt{max}\_\mathtt{nodes}\_\mathtt{per}\_\mathtt{level}]$ 16:         $\mathrm{iteration}\leftarrow \mathrm{iteration}+1$ 17: end while  18: Final Step: Select the prompt with the highest accuracy across all levels 19:   $P^{*}\leftarrow arg{max}_{(P_i,A_i)\in \mathtt{accuracy}\_\mathtt{list}}{A}_i$ x 20: return $P^{*}$

2.5. Selecting the Final Prompt

Once the maximum iteration is reached or no further FP/FN data are available, the expansion process stops. The training accuracy of all prompts across all iterations is recorded. The final step is comparing these accuracies. The prompt with the highest accuracy, when evaluated by $LL{M}_1$, is selected as the final prompt. If the initial prompt outperforms the newly generated prompts, the initial prompt will be chosen as the final prompt ($P^{*}$).

This final prompt constitutes the output of the proposed PO2G framework. In the following section, we illustrate the complete workflow of the PO2G framework, from receiving the initial prompt to producing the final optimized prompt.

2.6. PO2G Framework Illustration

This subsection details the process of refining a prompt from its initial version until reaching the final optimized prompt. Figure 6 and Figure 7 illustrate the PO2G framework over three iterations.

The PO2G framework begins by applying the initial prompt to the training data using $LL{M}_1$. The response generated by $LL{M}_1$ is collected as a prediction and subsequently used to generate feedback via either $LL{M}_{2FP}$ or $LL{M}_{2FN}$. The feedback is used to generate a new prompt using $LL{M}_3$. This iterative process of using $LL{M}_1$, $LL{M}_{2FP}$ or $LL{M}_{2FN}$, and $LL{M}_3$ is part of Algorithm 1 and is visually represented in Figure 6.

The process of expanding $P_0$ and selecting the final prompt is illustrated in Figure 7. The algorithm for prompt expansion and selection was previously discussed in Algorithm 3. After setting an initial prompt $P_0$, Algorithm 1 generates two new prompts: $P_{1.1}$ and $P_{1.2}$. These prompts are then evaluated using Algorithm 3, which determines whether a node should be marked as active or inactive. The iterative expansion of active nodes continues until a stopping criterion is met, resulting in the selection of the final prompt. As illustrated in Figure 7, the process concludes at iteration 3, with prompt $P_{3.4}$ selected as the final prompt, which achieves the highest training score accuracy of 91%.

2.7. Comparing the PO2G Framework with ProTeGi

The PO2G framework evaluates the initial prompt on the entire training dataset to obtain loss signals. In contrast, ProTeGi divides the data into random mini-batches and aggregates loss signals from each batch to refine the prompt. Additionally, while ProTeGi relies solely on random mini-batching, PO2G allows for either random sampling or clustering to select feedback.

A key difference is in how incorrect predictions are handled. The PO2G framework separates them into two distinct loss signals (one for false positives and one for false negatives) and clusters these signals into groups. In contrast, ProTeGi generates local loss signals by aggregating gradients from a minibatch’s data.

LLMs typically perform best with clear, targeted instructions; combining all feedback in one request can lead to poor responses or hallucinations. PO2G mitigates this by providing separate instructions for FP and FN data to minimize conflicting signals, ensuring more accurate feedback.

To refine the prompt further, ProTeGi generates paraphrases of the improved prompt to produce slight variations. In contrast, PO2G selects different samples, either randomly or via clustering, to explore alternative textual gradient directions rather than merely paraphrasing the text.

ProTeGi uses minibatch, whereas PO2G uses the entire dataset to assess the improved prompt when optimizing the prompt. This comprehensive evaluation helps avoid local optima that may arise from mini-batch sampling.

A potential drawback of using the entire training dataset is the risk of overfitting, as the prompt might be tailored too closely to the training data distribution. Although this approach increases computational time compared to mini-batch sampling, it reduces redundancy by avoiding repeated evaluations of the same prompt on identical data.

Each framework has its pros and cons, which we evaluate experimentally in terms of accuracy and computational efficiency. The results are discussed in the next section.

3. Experimental Setup

This section describes our experiment to evaluate the PO2G framework across 16 NLP tasks. These tasks fall into three categories: (i) the three tasks used in ProTeGi, (ii) six additional publicly available (non-domain) tasks, and (iii) seven domain-specific legal tasks.

Each task uses separate and balanced 200 training and 200 test samples, except for the legal NLP task, which is less than 200 and imbalanced. The legal tasks are smaller and imbalanced; for these, the training and test sets are identical (train=test) to study optimization behavior. The performance measure accuracy is evaluated on all tasks. For the imbalanced legal setting, performance metrics include positive-class precision, recall, and F₁.

3.1. Tasks and Data

The NLP classification tasks used in this study are listed below.

• Tasks used in ProTeGi research. [13]:

  –

  Ethos [25]

  –

  Liar [26]

  –

  Ar Sarcasm [27]

• Publicly available data (Non-Domain-Specific Tasks):

  –

  Financial Sentiment [28]

  –

  Amazon Review [29]

  –

  Tweet Airline [30]

  –

  Tweet Hate Speech [31]

  –

  Tweet Offensive Language [31]

  –

  Clickbait [32]

• Legal document data (Domain-Specific Tasks):

  –

  Italian FTT documents (150 samples):

  ∗

  Tax Product

  ∗

  Tax Transaction

  ∗

  Tax Exemption

  ∗

  Tax Calculation

  ∗

  Tax Subject

  ∗

  Tax Process

  –

  FCA Regulation Document:

  ∗

  Obligation Extraction

3.2. Language Models and API Configuration

All of the main experiments use GPT-4o Mini (gpt-4o-mini-2024-07-18) across frameworks to ensure fairness. PO2G relies on parallel LLM calls; running comparable open-source models would require hardware we did not have available, so we restrict evaluation to OpenAI-served models provided by our sponsor. On ablation (Section 4.3), we swap the feedback/prompt-generation components ($LL{M}_{2FP}$, $LL{M}_{2FN}$, and $LL{M}_3$) to GPT-4o, while keeping the classifier $LL{M}_1$ as GPT-4o Mini; this isolates the effect of a stronger LLM under a fixed classifier.

3.3. Prompt Initialization

A consistent initial prompt is used throughout all experiments. For the three ProTeGi tasks, we reuse the original ProTeGi prompts; for the remaining tasks, we construct comparable, concise instructions aligned with the ProTeGi style (Appendix 

Table A3. Initial prompts used at iteration 0.

Task ID	Task	Initial Prompt
Category I
9.1	ethos	Is the following text hate speech?
10.1	liar	Determine whether the statement is a lie (Yes) or not (No) based on the context and other information.
11.1	ArSarcasm	Is this tweet sarcastic?
Category II
3.1	Financial Sentiment	Is this text expressing a positive overall sentiment?
4.1	Amazon_review	Is this text a positive review?
5.1	Tweet_airline	Is this text containing a positive meaning?
6.1	HateSpeech	Is this text hate speech?
7.1	Offensive_Language	Is this text offensive language?
8.1	ClickBait	Is this text clickbait?
Category III
1.1	tax_product	Is this text about a tax product that is taxed by the authority?
1.2	tax_transaction	Is this text about a tax transaction?
1.3	tax_exemption	Is this a tax exemption?
1.4	tax_calculation	Is this a method or formula for calculating tax?
1.5	tax_subject	Is this text about someone who is liable to pay or report tax?
1.6	tax_process	Is this a process of paying tax to the authority?
2.1	Obligation	Is this text an obligation?

). While most experiments initialize from these prompts, the ablation study also evaluates an empty initialization to test the framework’s ability to discover prompts from scratch.

3.4. Systems Compared

Our goal is a faster, sample-efficient prompt optimizer, with special attention to legal annotation. We evaluate the following:

• ProTeGi (baseline): We reimplement ProTeGi [13] with identical algorithmic settings (e.g., beam size $=5$) under GPT-4o Mini.
• PO2G+C (proposed with clustering): PO2G uses two loss signals (one from false positives (FP) and one from false negatives (FN)) to propose and score edits during each iteration. We cluster the FP and FN error sets separately (default clusters $k=5$ per side) and select the most influential instances within clusters to form gradients. We then expand each gradient into candidate edits (max expansion $=2$) and score them.
• PO2G (no clustering): This is identical to PO2G+C but without clustering. FP and FN examples are sampled randomly (default random samples $=5$ per side), expanded (max expansion $=2$), and scored.

3.5. Evaluation Organization

• Non-domain performance: Accuracy and cumulative API calls across nine tasks (Section 1.1); robustness calls on three tasks (LIAR, AR_Sarcasm, and Clickbait) with five independent runs each, reporting SE and pairwise t-test.
• Legal (domain) behavioral analysis: Accuracy, API calls, and positive-class precision/recall/F₁ across seven tasks with train=test; robustness on Obligation with five runs, reporting SE and pairwise t-test.
• Ablations: Clustering (PO2G+C vs. PO2G), empty vs. initial prompt, and LLM by mainly comparing the accuracy and API calls.

3.6. Iteration Budget, Metrics, and Statistics

Optimization proceeds for six self-refinement iterations ($i\in \{0,\dots ,6\}$), where $i=0$ denotes the evaluation of the initial prompt. We report on the following:

• Accuracy and cumulative API calls per iteration.
• Positive-class precision/recall/F₁ for legal imbalance data.
• Robustness: five independent runs for selected tasks; we report the mean and standard error (SE = SD$/\sqrt{5}$).
• The pairwise t-test: comparing frameworks’ accuracy on a certain condition explained in the results and discussion. Exact p-values are reported.

Prompts are provided in the Appendix. All non-domain experiments use different train/test data of size 200 each. Legal tasks are smaller and imbalanced; we use train=test to study optimization behavior due to data limitations. Robustness runs use the identical configuration five times; the difference in performance at different times is be related to LLM stochasticity.

4. Results and Discussion

This section presents accuracy and cumulative API calls. We separate the analysis into non-domain datasets for the main performance comparison, a legal-domain behavioral analysis, and three ablation studies. Following prior work on ProTeGi, accuracy is the primary metric. An exploratory study on imbalanced data will report precision, recall, and F₁ for the positive class, as explained in the previous section.

4.1. Main Performance on Non-Domain Tasks

To compare prompt-optimization strategies, we evaluate both the achieved accuracy and API calls used during optimization. We first compare iterations 0, 3, and 6 across nine datasets, then analyze the accuracy and cumulative API calls over all iterations. Finally, we report robustness (SE) and per-dataset pairwise t-test on three datasets with five repeated runs.

4.1.1. Overview

Initial Prompt (Iteration 0)

Table 4. Non-domain initial performance comparison of ProTeGi and PO2G+C across multiple tasks.

Task ID	Task	ProTeGi		PO2G+C
		Accuracy	API Calls	Accuracy	API Calls
9.1	Ethos	0.9350	200	0.9350	200
10.1	Liar	0.5020	200	0.5280	200
11.1	ARSarcasm	0.8270	200	0.8410	200
Average_1		0.7547	200	0.7680	200
3.1	Sentiment	0.9350	200	0.9400	200
4.1	Amazon Review	0.9300	200	0.9300	200
5.1	Tweet Airline	0.9400	200	0.9400	200
6.1	Hate Speech	0.9000	200	0.9000	200
7.1	Offensive Language	0.8550	200	0.8550	200
8.1	Clickbait	0.8630	200	0.8890	200
Average_2		0.9038	200	0.9090	200
Average_all		0.8541	200	0.8620	200

reports on the accuracy and API calls using the same initial prompt and identical LLM/decoding settings. As expected, both frameworks require 200 API calls (one per training instance) and achieve similar accuracy; small differences remain despite identical configurations, reflecting LLM stochasticity.

Iteration 3

After three refinements (

Table 5. Non-domain performance comparison of ProTeGi and PO2G+C across multiple tasks for iteration 3.

Task ID	Task	ProTeGi		PO2G+C
		Accuracy	API Calls	Accuracy	API Calls
9.1	Ethos	0.9350	7481	0.9300	6462
10.1	Liar	0.5820	7344	0.6130	10,300
11.1	ARSarcasm	0.8280	7510	0.8670	10,138
Average_1		0.7817	7445	0.8033	8967
3.1	Sentiment	0.9850	6989	0.9950	4240
4.1	Amazon Review	0.9450	7080	0.9400	3230
5.1	Tweet Airline	0.9450	7118	0.9550	8886
6.1	Hate Speech	0.8850	7055	0.9000	10,098
7.1	Offensive Language	0.8600	6822	0.8650	10,098
8.1	Clickbait	0.8980	6779	0.9270	9613
Average_2		0.9197	6974	0.9303	7694
Average_all		0.8737	7131	0.8880	8118

), the average accuracy across nine datasets diverges: PO2G+C attains a higher mean accuracy than ProTeGi, typically with a higher API-call budget, though several high-accuracy tasks (e.g., Ethos, Sentiment, and Amazon Review) show fewer calls for PO2G+C. A pairwise t-test across datasets at Iteration 3 yields $p=0.0256$ (two-sided), indicating a statistically significant mean difference in accuracy under our protocol.

Iteration 6

By six iterations (

Table 6. Non-domain performance comparison of ProTeGi and PO2G+C across multiple tasks during iteration 6.

Task ID	Task	ProTeGi		PO2G+C
		Accuracy	API Calls	Accuracy	API Calls
9.1	Ethos	0.9500	15,070	0.9250	13,128
10.1	Liar	0.6020	14,998	0.6010	22,420
11.1	ARSarcasm	0.8440	15,029	0.8650	22,258
Average_1		0.7987	15,032	0.7970	19,269
3.1	Sentiment	0.9900	14,401	0.9400	7876
4.1	Amazon Review	0.9400	14,478	0.9500	6462
5.1	Tweet Airline	0.9250	14,469	0.9400	19,794
6.1	Hate Speech	0.9100	14,468	0.9150	22,016
7.1	Offensive Language	0.8650	14,012	0.9350	20,400
8.1	Clickbait	0.9420	14,144	0.9350	20,400
Average_2		0.9287	14,329	0.9358	16,158
Average_all		0.8853	14,563	0.8896	17,195

), the accuracy gap narrows and both frameworks approach a plateau, suggesting diminishing returns and a possible LLM ceiling. Mean accuracy for PO2G+C remains slightly higher, but a The pairwise t-test at iteration 6 is not significant ($p=0.7110$). API cost continues to diverge: ProTeGi averages ∼14–15 thousand calls, whereas PO2G+C exceeds 17,000 on average during iteration 6, reflecting its more expensive approach per iteration.

Conducting a The pairwise t-test on the ProTegi@6 vs. PO2G@3 iteration step, the result is 0.6451, which is statistically not significant. The average accuracy of PO2G in three iterations was still higher compared to ProTeGi in six iterations. In terms of overall performance comparison, it suggests that the performance of PO2G@3 is similar to that of ProTeGi@6 while requiring fewer API calls to achieve a similar level of performance.

4.1.2. Accuracy

Figure 8 and

Table 7. Average performance of the non-domain dataset per iteration.

Iteration	ProTeGi		PO2G+C
	Accuracy	API Calls	Accuracy	API Calls
0	0.8541	200	0.8620	200
1	0.8694	2347	0.8678	1843
2	0.8773	4681	0.8858	5039
3	0.8737	7131	0.8880	8118
4	0.8686	9599	0.8921	11,171
5	0.8777	12,084	0.8902	14,165
6	0.8853	14,563	0.8896	17,195

show iteration-wise averages of non-domain data, for which accuracy is the primary metric. Accuracy generally increases from iteration 0 to 6. ProTeGi exhibits mild fluctuations (iterations 2–4), consistent with minibatch-based selection and bandit scoring; PO2G+C shows smoother gains. The fluctuation in performance may be attributed to the use of minibatches, suggesting that ProTeGi, during iterations 3 to 4, is overfitting to a specific local minibatch, which results in a higher prompt score compared to other prompts tested across multiple minibatches. Further research are needed to confirm the effect of using Minibatch.

4.1.3. API Call Efficiency and Trade-Off

Cumulative API calls increase with iteration depth for both methods (Figure 9 and Table 7). Under the default settings, PO2G+C typically incurs a higher call budget than ProTeGi at the same iteration on the non-domain average (e.g., 17,194.9 vs. 14,563.2 at iteration 6). The gap is explained by the optimization process: PO2G+C harvests feedback from both false positives and false negatives and evaluates a larger pool of candidate edits, whereas ProTeGi’s bandit/minibatch scoring constrains evaluations per iteration per mini batch. When accuracy becomes high and incorrect predictions become scarce, the number of feedback edits to the prompt that can be proposed is reduced. This is why some iteration 3 tasks exhibit fewer calls for PO2G+C, despite usually having higher costs.

4.1.4. Statistical Significance on Three Public Datasets (5 runs)

We run five independent trials on LIAR, ARSarcasm, and Clickbait, reporting per-dataset SE (standard error) and The pairwise t-test for iterations 3 and 6. On LIAR, PO2G+C significantly outperforms ProTeGi at $i=3$ ($p=0.0280$), but not at $i=6$ ($p=0.7833$). For ARSarcasm, the difference is not significant at $i=3$ ($p=0.1254$) but becomes significant at $i=6$ ($p=0.0173$). For Clickbait, differences are not significant during either iteration ($p=0.2796$, $p=0.5734$). Full run-level results are presented in

Table 8. Performance comparison of ProTeGi and PO2G+C on the LIAR dataset during iterations 3 and 6.

Run	Iteration = 3		Iteration = 6
	ProTeGi	PO2G+C	ProTeGi	PO2G+C
1	0.5900	0.6250	0.6150	0.6250
2	0.5900	0.6250	0.6150	0.6250
3	0.6000	0.6300	0.6200	0.6400
4	0.5900	0.6050	0.5900	0.5600
5	0.5800	0.5800	0.6100	0.5850
Average	0.5900	0.6130	0.6100	0.6070
SE	0.0032	0.0093	0.0052	0.0149
Pairwise t-test	0.0280		0.7833

,

Table 9. Performance comparison of ProTeGi and PO2G+C on ARSARCASM dataset during iterations 3 and 6.

Run	Iteration = 3		Iteration = 6
	ProTeGi	PO2G+C	ProTeGi	PO2G+C
1	0.8550	0.8550	0.8450	0.8650
2	0.8550	0.8550	0.8450	0.8650
3	0.7900	0.8800	0.8100	0.8500
4	0.8250	0.8650	0.8500	0.8550
5	0.8500	0.8800	0.8500	0.8750
Average	0.8350	0.8670	0.8400	0.8620
SE	0.0125	0.0056	0.0076	0.0044
Pairwise t-test	0.1254		0.0173

and

Table 10. Performance comparison of ProTeGi and PO2G+C on the CLICKBAIT dataset during iterations 3 and 6.

Run	Iteration = 3		Iteration = 6
	ProTeGi	PO2G+C	ProTeGi	PO2G+C
1	0.8250	0.9750	0.9400	0.9550
2	0.8250	0.9750	0.9400	0.9550
3	0.9200	0.9200	0.9500	0.9150
4	0.9550	0.9200	0.9450	0.9400
5	0.9300	0.9200	0.9550	0.9350
Average	0.8910	0.9420	0.9460	0.9400
SE	0.0275	0.0135	0.0029	0.0074
Pairwise t-test	0.2796		0.5734

.

The results of the The pairwise t-test and SE comparing ProTeGi and PO2G are inconclusive. In the next subsection, their performance on legal data is displayed.

4.2. Legal Data Behavioral Analysis

4.2.1. Overview

As the primary purpose of the proposed framework is document annotation, we include an exploratory analysis on imbalanced legal datasets provided by our sponsor. These data are reused for optimization and evaluation (train=test), so the goal is to study behavior. The legal datasets are annotated by domain experts. Tax-related data involve element annotation from Italian FTT documents; the Obligation data derive from FCA regulation annotations.

Initial Prompt (Iteration 0)

Using the same prompt and LLM, accuracies are very similar (

Table 11. Legal domain initial performance comparison of ProTeGi and PO2G+C across multiple tasks.

Task ID	Task	ProTeGi		PO2G+C
		Accuracy	API Calls	Accuracy	API Calls
1.1	Tax Product	0.5200	150	0.5867	150
1.2	Tax Transaction	0.3600	150	0.3600	150
1.3	Tax Exemption	0.9600	150	0.9667	150
1.4	Tax Calculation	0.9667	150	0.9667	150
1.5	Tax Subject	0.5800	150	0.6000	150
1.6	Tax Process	0.5667	150	0.5800	150
2.1	Obligation	0.4670	200	0.4670	200
Average		0.6315	157	0.6467	157

); small deviations reflect LLM stochasticity.

Iteration 3

Both frameworks improve substantially by iteration 3; PO2G+C attains a higher average accuracy and a slightly lower average API budget (

Table 12. Legal domain performance comparison of ProTeGi and PO2G+C across multiple tasks during iteration 3.

Task ID	Task	ProTeGi		PO2G+C
		Accuracy	API Calls	Accuracy	API Calls
1.1	Tax Product	0.9400	6366	0.9800	6382
1.2	Tax Transaction	0.8533	6440	0.9267	5926
1.3	Tax Exemption	0.9733	6068	0.9800	1974
1.4	Tax Calculation	0.9933	6417	1.0000	3646
1.5	Tax Subject	0.8000	6438	0.8733	6838
1.6	Tax Process	0.8467	6383	0.8600	7598
2.1	Obligation	0.8690	7308	0.8950	9452
Average		0.8965	6489	0.9307	5974

), indicating a favorable accuracy–cost trade-off.

Iteration 6

By iteration 6, both frameworks continue to improve, with PO2G+C maintaining an accuracy advantage and a lower average API budget overall (

Table 13. Legal domain performance comparison of ProTeGi and PO2G+C across multiple tasks during iteration 6.

Task ID	Task	ProTeGi		PO2G+C
		Accuracy	API Calls	Accuracy	API Calls
1.1	Tax Product	0.9533	13,575	0.9800	11,398
1.2	Tax Transaction	0.8933	13,640	0.9267	13,982
1.3	Tax Exemption	0.9867	13,201	0.9800	3950
1.4	Tax Calculation	0.9933	13,593	1.0000	7142
1.5	Tax Subject	0.8133	13,565	0.9000	15,958
1.6	Tax Process	0.8467	13,504	0.8733	18,300
2.1	Obligation	0.8740	14,688	0.9240	20,642
Average		0.9087	13,681	0.9406	13,053

).

Overall, by iteration 6, PO2G+C has higher mean accuracy and, on average, is more efficient (fewer cumulative API calls) than ProTeGi in these legal annotation tasks. The per-task costs are heterogeneous and API savings concentrate where PO2G+C converges quickly (e.g., Tax Product, Tax Exemption, and Tax Calculation), while other tasks still incur higher call budgets. Because the datasets are imbalanced, we further report positive-class precision, recall, and F₁ in Section 4.2.4 to validate these performance gains.

4.2.2. Accuracy

Figure 10 and

Table 14. Average performance of the legal domain dataset per iteration.

Iteration	ProTeGi		PO2G+C
	Accuracy	API Calls	Accuracy	API Calls
0	0.6315	157	0.6467	157
1	0.8659	1847	0.8929	1165
2	0.8803	4094	0.9154	3532
3	0.8965	6489	0.9307	5974
4	0.9013	8889	0.9337	8876
5	0.9061	11,283	0.9373	11,025
6	0.9087	13,681	0.9406	13,053

show iteration-wise accuracy concerning legal tasks. Both frameworks improve substantially from the initial iteration, confirming the benefit of iterative prompt refinement. PO2G+C consistently outperforms ProTeGi from iteration 1 onward and reaches its peak performance faster, indicating faster optimization of the observed training pool.

4.2.3. API Call Efficiency and Trade-Off

Figure 11 and Table 14 illustrate average cumulative API calls per iteration concerning the legal tasks. Call counts increase monotonically for both methods, but PO2G+C consistently requires fewer calls than ProTeGi during the same iteration (e.g., 13,053 vs. 13,681 at $i=6$). This pattern reflects optimization dynamics: as PO2G+C reaches higher accuracy earlier, false positives/negatives become scarce, yielding fewer actionable feedback items and thus fewer candidate edits to generate and evaluate. In contrast, ProTeGi’s bandit/minibatch scoring continues to evaluate a broader set of candidates per round.

4.2.4. Precision, Recall, and F1 Score

Table 15. Precision, recall, and F₁ per iteration on the legal domain task.

Iter.	ProTeGi			PO2G+C
	Prec.	Rec.	F1	Prec.	Rec.	F1
0	0.5248	0.8078	0.5389	0.5575	0.7747	0.5422
1	0.6353	0.5332	0.5612	0.7995	0.6379	0.6961
2	0.6595	0.6364	0.6439	0.8740	0.7217	0.7643
3	0.7771	0.7520	0.7516	0.9174	0.7950	0.8137
4	0.7998	0.7440	0.7693	0.8839	0.8172	0.8367
5	0.8039	0.7484	0.7587	0.8960	0.8194	0.8409
6	0.8265	0.7591	0.7754	0.9218	0.7945	0.8401

reports positive-class precision, recall, and F₁ across iterations concerning the legal tasks. Iteration 0 uses identical prompts but shows slight performance differences, which we attribute to LLM stochasticity. We focus on the positive label when carrying out document annotation. Both frameworks improve steadily with each iteration; however, from iteration 3 onward, PO2G+C exhibits a more rapid and consistent rise in all three metrics. By the final iteration, PO2G+C attains the highest F₁ (0.8401), along with precision and recall, indicating more reliable detection of positive instances. These results indicate that PO2G+C provides a sustained advantage on the positive class recall precision, which translates into higher F₁ over six iteration steps.

4.2.5. Statistical Significance

Table 16. Performance comparison of ProTeGi and PO2G+C on the Obligation dataset during iterations 3 and 6.

Run	Iteration = 3		Iteration = 6
	ProTeGi	PO2G+C	ProTeGi	PO2G+C
1	0.8600	0.8700	0.8750	0.9200
2	0.8600	0.8700	0.8750	0.9200
3	0.8900	0.9000	0.8850	0.9250
4	0.9050	0.9000	0.8500	0.9100
5	0.8350	0.9050	0.8850	0.9400
Average	0.8700	0.8890	0.8740	0.9230
SE	0.0123	0.0078	0.0064	0.0049
Pairwise t-test	0.2199		0.0002

compares ProTeGi and PO2G+C on the Obligation dataset over five independent runs during iterations 3 and 6. Both methods improve between iterations, and PO2G+C has a higher mean accuracy at each stage. The pairwise t-test indicate that the difference during iteration 3 is not significant ($p=0.2199$), whereas by iteration 6, it becomes highly significant ($p=0.0002$). Standard errors are lower for PO2G+C at both iterations, suggesting slightly better run-to-run stability on this condition.

While most of the performance metrics favor PO2G+C in the legal domain test, note that the legal setting here uses train=test for behavioral analysis; all results in this subsection are an exploration to characterize optimization on the observed pool and do not establish external generalization. Stronger claims in the legal domain will require held-out evaluations on valid data. In the following subsection, we present an ablation study.

4.3. Ablation Study

This section presents an ablation study conducted to assess the influence of three design choices on the proposed framework: clustering, the presence/absence of an initial prompt, and LLM choice.

4.3.1. Clustering

We employ clustering to reduce redundancy by selecting representative centroids and prioritizing influential samples, aiming to enhance efficiency. The following tables compare the two PO2G variants (with and without clustering).

Across the nine non-domain datasets (

Table 17. Non-domain tasks: PO2G baseline and differences for PO2G+C relative to PO2G across iterations. Positive $\Delta$ accuracy favors PO2G+C; negative $\Delta$ API calls indicate fewer calls for PO2G+C.

Iteration	PO2G		$\Delta$ (PO2G+C − PO2G)
	Accuracy	API Calls	Accuracy	API Calls
0	0.8552	200	0.0068	0
1	0.8712	1888	−0.0034	−45
2	0.8760	5035	0.0098	4
3	0.8892	8006	−0.0012	112
4	0.8961	11,216	−0.0040	−45
5	0.8932	14,398	−0.0030	−233
6	0.8853	19,062	0.0042	−1867

), clustering (+C) generally reduces the long-run API budget while maintaining very similar accuracy. On average, at iteration 6, PO2G+C lowers cumulative API calls by ≈1.8 thousand relative to PO2G with minimal performance difference that should be LLM stochasticity.

On the legal (domain) data where train=test, adding clustering improves average accuracy across all seven tasks and tends to reduce calls during later iterations (

Table 18. Legal domain tasks: PO2G baseline and differences for PO2G+C relative to PO2G across iterations. Positive $\Delta$ accuracy favors PO2G+C; negative $\Delta$ API calls indicate fewer calls for PO2G+C.

Iteration	PO2G		$\Delta$ (PO2G+C − PO2G)
	Accuracy	API Calls	Accuracy	API Calls
0	0.6423	157	0.0044	0
1	0.8781	1199	0.0149	−33
2	0.8957	3484	0.0197	48
3	0.8977	6073	0.0330	−100
4	0.9033	8041	0.0304	835
5	0.9113	11,776	0.0259	−751
6	0.9201	14,599	0.0205	−1546

).

Across the domain tasks in

Table 19. Domain tasks: PO2G baseline and differences for PO2G+C relative to PO2G across iterations. Positive $\Delta$ favors PO2G+C.

Iteration	PO2G			$\Delta$ (PO2G+C − PO2G)
	Precision	Recall	F1	$\Delta$Prec.	$\Delta$Rec.	$\Delta$F1
0	0.5623	0.7725	0.5396	$-0.0049$	0.0023	0.0026
1	0.7455	0.6541	0.6897	0.0541	$-0.0162$	0.0064
2	0.7502	0.6576	0.6625	0.1239	0.0641	0.1018
3	0.7581	0.6492	0.6627	0.1593	0.1459	0.1511
4	0.8281	0.7225	0.7331	0.0557	0.0948	0.1036
5	0.8735	0.7192	0.7504	0.0225	0.1002	0.0905
6	0.8840	0.7373	0.7656	0.0379	0.0571	0.0744

, PO2G+C consistently outperforms PO2G on positive-class precision, recall, and F₁ once refinement is underway. Differences are negligible during iteration 0 and mixed during iteration 1, after which PO2G+C yields clear improvements and remains positive throughout iteration 6. This pattern suggests that clustering strengthens minority-class detection by raising both recall and precision.

Concerning non-domain data, clustering delivers modest accuracy changes but a better long-run accuracy–cost trade-off; during the (train=test) domain analysis, it improves both accuracy and, throughout later iterations, the call budget. Effects likely depend on the batch size and pool size; we do not evaluate clustering at larger training scales, leaving this to future studies.

4.3.2. Empty Prompt

Here, we remove the initial prompt entirely, using an empty starting point. Experiments are run on Obligation, LIAR, AR_Sarcasm, and Clickbait (reusing the datasets from Section 4.1.4), showing performance in the

Table 20. Empty vs. Initial prompt iterations ($i=0,3,6$). $\Delta$Acc = Initial − Empty (positive value favors the Initial prompt). Initial-prompt accuracies are reported in Table 14.

Dataset	Empty Prompt (Test Accuracy.)			$\mathbf{\Delta }$Accuracy (Initial − Empty)
	$i=\mathbf{0}$	$i=\mathbf{3}$	$i=\mathbf{6}$	$i=\mathbf{0}$	$i=\mathbf{3}$	$i=\mathbf{6}$
LIAR	0.3650	0.6800	0.6200	0.1630	−0.0670	−0.0190
AR_Sarcasm	0.3800	0.8950	0.9000	0.4610	−0.0280	−0.0350
Clickbait	0.4250	0.6250	0.6600	0.4640	0.3020	0.2750
Obligation	0.6300	0.7900	0.8000	−0.1630	0.1050	0.1240
Average	0.4500	0.7475	0.7450	0.2313	0.0780	0.0863

. During iteration 0, an empty prompt should be random. Adding a good initial prompt acts as a strong prior and typically accelerates optimization; however, we observe mixed outcomes. Clickbait and Obligation benefit from the initial prompt by iterations 3 and 6, while LIAR and AR_Sarcasm perform better from an empty start, suggesting the provided initial prompt was suboptimal for these datasets and may have caused the optimizer to converge on less helpful gradients.

We do not pursue further analysis here, but the results illustrate that the optimizer can recover useful prompts even from an empty start, potentially requiring more iterations than having a proper initial prompt.

4.3.3. LLM Comparison

In this subsection, we analyze the impact of different LLM configurations on model performance as part of the ablation study. We compare the GPT-4o and cheaper 4o Mini across four datasets: Liar, AR_sarcasm, Clickbait, and Obligation.

Table 21. Comparison of GPT-4o performance and its delta to 4o-mini–GPT-4o across key iterations (i = 0, i = 3, and i = 6). Positive $\Delta$ favors 4o-mini.

Dataset	GPT-4o			$\Delta$ 4o-mini - GPT-4o
	i = 0	i = 3	i = 6	i = 0	i = 3	i = 6
LIAR	0.5050	0.6000	0.5800	0.0230	0.0130	0.0210
AR_Sarcasm	0.8500	0.8700	0.8650	−0.0090	−0.0030	0.0000
Clickbait	0.8850	0.9800	0.9900	0.0040	−0.0530	−0.0550
Obligation	0.4650	0.8750	0.8950	0.0020	0.0200	0.0290

presents the results, focusing on test accuracy. We change the LLM on $LL{M}_{2FP}$, $LL{M}_{2FN}$, and $LL{M}_3$ to larger LLMs, resulting in an increase and decrease in accuracy. There is quite a significant increase in performance on the Clickbait dataset; however, there is a decrease in performance on the Liar and Obligation datasets.

We select a sample of prompts and examine the length of the final optimized prompts at i = 6 in

Table 22. Sample prompt length (word count) at iteration 6 for two LLMs.

Dataset	GPT-4o	4o Mini
LIAR	360	222
AR_Sarcasm	466	67
Clickbait	459	302
Obligation	440	31

. GPT-4o Mini generally produces prompts with less length and detail compared to GPT-4o. Comparing performance to the length of the prompt with numerous details or samples does not guarantee a better result. However, it increases the inference cost.

Our results are mixed. Using GPT-4o for feedback/prompt generation improves Clickbait but reduces LIAR and Obligation relative to GPT-4o Mini, possibly due to a mismatch between the LLM (GPT-4o) and the classifier LLM ($LL{M}_1=$4o-Mini). Longer, detailed prompts from GPT-4o do not guarantee better accuracy and will increase inference cost. A full test on other LLM configurations is left to future work due to budget and hardware limitations. The instructions generated by GPT-4o may not be fully understood by 4o-Mini due to its smaller model and lower capability compared to GPT-4o.

5. Conclusions

In this paper, we introduced PO2G+C, an automatic and discrete prompt optimization framework. Our experiments demonstrated that PO2G outperforms the ProTeGi framework on the same iteration step. Importantly, the PO2G framework reaches the performance of ProTeGi@6 by iteration 3 on average non-domain datasets, as shown in Section 4.1. Although PO2G+C typically incurs a higher per-iteration call budget, it attains a comparable accuracy in fewer iterations. The total accumulated calls to reach a target accuracy can be lower (e.g., PO2G+C@3 vs. ProTeGi@6). We also conduct a pairwise t-test and SE to check the statistical significance across nine datasets. The pairwise t-test on PO2G@3 was statistically significant against ProTeGi@3 and was not statistically significant with ProTeGi@6.

On legal (domain) data, where train = test, PO2G+C shows higher accuracy, precision, recall, and F1 than ProTeGi while using fewer or comparable cumulative API calls by later iterations. Because these experiments evaluate optimization behavior rather than external generalization, we treat them as a behavioral analysis instead of claiming performance improvement.

Our ablation studies yield three takeaways.

• Clustering generally reduces long-run API calls with minimal accuracy change on non-domain data, and it improves accuracy and reduces cost on legal tasks; we hypothesize larger gains with bigger training pools for future work.
• Initial prompt quality matters: a good seed can accelerate optimization, whereas a suboptimal seed can mislead the search; nevertheless, PO2G can generate a prompt from an empty prompt, albeit typically with more iterations.
• LLM choice matters: using GPT-4o for feedback/prompt generation produces longer, more detailed prompts and mixed accuracy changes when the classifier remains 4o-mini, suggesting a mismatch effect; fully matched upgrades (generator and classifier) remain to be assessed and are left for future work.

6. Limitations

• Legal domain

Legal experiments use small, imbalanced datasets and train on the same data as the test set. These settings are useful for behavioral analysis of optimization dynamics, but not valid as a proper benchmark to claim performance improvement. Reported gains used as behavior analysis. A proper legal benchmark is needed for benchmark validity.

• LLM scope

Due to resource constraints and the framework’s reliance on parallel API calls, we primarily evaluated it with OpenAI GPT-4o-mini, conducting ablations that utilize GPT-4o for feedback/prompt generation, while maintaining the classifier as 4o-mini. This generator–classifier mismatch can influence results and was not exhaustively explored. We did not test open-source LLMs due to hardware limitations; performance and cost may vary under different providers, model sizes, or settings.

• Data size

Following ProTeGi [13], we conducted experiments with 200 train/200 test examples per dataset. While the PO2G framework helps avoid local overfitting in minibatches, it still overfits to the training data over deeper iterations. Increasing the dataset size could further mitigate overfitting, though at the expense of additional testing time. Notably, expanding the optimization iterations from three to six doubled the computational effort but only yielded marginal performance improvements, suggesting that the optimal iteration count depends on both dataset size and the specific NLP task before diminishing returns.

• Cost measurement

We primarily reported API call counts as a proxy for the computational cost. In depth accounting (input/output token usage, monetary price, etc.) was out of scope. This may alter the perceived efficiency trade-offs, especially when prompts become longer using more expensive LLMs.

• Statistics and robustness

Robustness analyses used n = 5 runs per dataset. Pairwise t-test were reported per dataset and iteration, but no clear conclusion was drawn regarding robustness or statistical significance across use cases. Despite identical initial prompts and settings, repeating PO2G runs exhibited variation due to LLM stochasticity, which affects robustness and reproducibility.

• Classification task

Since the framework was designed for document annotation, it was evaluated as a classification task using FP and FN For non-classification tasks; therefore, alternative suitable metrics (other than FP and FN) are needed to guide prompt optimization. Further exploration of loss signal categorization may also enhance this process.