Chain-of-Thought Prompt Optimization via Adversarial Learning

Abstract

Chain-of-Thought (CoT) prompting has demonstrated strong effectiveness in improving the reasoning capabilities of Large Language Models (LLMs). However, existing CoT optimization approaches still lack systematic mechanisms for evaluating and refining prompts. To address this gap, we propose Adversarial Chain-of-Thought (adv-CoT), a framework that introduces adversarial learning into prompt optimization. Adv-CoT iteratively refines an initial prompt through generator–discriminator interactions and integrates both feedback and verification mechanisms. This process enables more targeted and interpretable improvements to CoT instructions and demonstrations. We evaluate adv-CoT on twelve datasets across commonsense, factual, symbolic, and arithmetic reasoning. Across 12 reasoning datasets, adv-CoT yields an average improvement of 4.44% on GPT-3.5-turbo and 1.08% on GPT-4o-mini, with both gains being statistically significant (paired t-test, p < 0.05). The experimental results show that the framework yields consistent but task-dependent gains, particularly on numerical and factual reasoning tasks, and maintains competitive performance on symbolic and commonsense benchmarks. Paired significance tests further indicate that improvements are statistically reliable on high-capacity proprietary models, while results on smaller open-source models exhibit greater variance. Although these findings demonstrate the promise of adversarial refinement for CoT prompting, the conclusions remain preliminary. The effectiveness of adv-CoT depends on the base model’s reasoning capability, and the current evaluation is limited to four major categories of reasoning tasks. We will release the full implementation and prompts to support further investigation into broader applications and more generalizable prompt optimization strategies.

1. Introduction

Large Language Models (LLMs) have achieved significant advances in Natural Language Processing (NLP) tasks [1,2,3,4,5]. To better guide their reasoning process, researchers have introduced In-Context Learning (ICL) [6,7,8]. Moreover, Chain-of-Thought (CoT) [9,10,11,12,13] prompting has demonstrated significant effectiveness in enhancing the performance of LLMs. Unlike traditional prompts that require the model to directly produce an answer, often resulting in abrupt reasoning and hallucinations [14,15,16,17], causing logical errors, CoT encourages the model to articulate intermediate reasoning steps. By decomposing complex problems into smaller subtasks, this approach facilitates error tracing and substantially improves output accuracy, particularly in domains such as mathematical and symbolic reasoning.

In traditional CoT, the reasoning process generated by LLMs is often difficult to effectively verify for authenticity [18]. Moreover, CoT prompts typically require task-specific adjustments for different application scenarios, lacking generality and scalability [13,19,20]. Similarly, despite the rapid progress of CoT prompting, existing approaches still lack a principled and scalable mechanism for optimizing and verifying CoT prompts. Current methods typically rely on heuristic refinement, manual prompt editing, or sampling-based Self-Consistency, all of which provide limited control over error propagation and lack explicit criteria for assessing the quality of intermediate reasoning steps. Furthermore, most prior work focuses on improving the final answer accuracy rather than systematically evaluating whether the reasoning process itself becomes more reliable, diverse, or robust. Existing approaches [13,21,22,23,24] still rely heavily on costly downstream evaluations and lack a principled optimization framework for prompt design. These limitations motivate us to explore a more efficient and theoretically grounded approach.

Generative Adversarial Networks (GANs) and adversarial learning [25] have achieved remarkable success in various domains, including image generation [25,26] and model robustness enhancement [27,28]. At their core, GANs consist of a generator-discriminator framework where the generator aims to increase the likelihood of the discriminator making errors by producing data indistinguishable from real samples, while the discriminator seeks to correctly differentiate between generated and authentic data. Throughout training, both components evolve adversarially.

Inspired by GANs and Adversarial In-Context Learning (adv-ICL) [29], we introduce adversarial learning into the evaluation and optimization of CoT prompts. We propose the Adversarial Chain-of-Thought (adv-CoT) framework, which takes an initial prompt as input and, through several rounds of iterative refinement, produces two fully developed sets of CoT-based instructions and demonstrations, with one serving as the generator and the other as the discriminator. Without modifying model parameters, adv-CoT enhances LLM performance by optimizing prompts adversarially. Compared with existing optimization approaches, our method introduces an adversarially guided refinement loop that exposes model-specific reasoning failures in a targeted manner. adv-CoT actively modifies the underlying instructions and demonstrations. In contrast to verification-based approaches that evaluate final answers, our framework incorporates verification into the optimization loop itself, providing a more principled mechanism for improving both stability and reliability. The comparison between the traditional CoT method and our adv-CoT is illustrated in Figure 1.

To guide the development of our adversarial Chain-of-Thought framework, we focus on three core research questions:

RQ1.

How can we design a principled and scalable mechanism to refine CoT prompts without model fine-tuning, enabling iterative improvement through structured adversarial feedback?

RQ2.

Can adversarial interactions between generator and discriminator expose weaknesses in reasoning chains and drive more accurate, stable, and error-resistant CoT reasoning across diverse datasets and task types?

RQ3.

How can explicit verification signals be integrated into the optimization loop to evaluate and enhance the reliability of the generated reasoning paths and final answers?

These RQs directly shape the design of our adv-CoT framework and the corresponding analyses in later sections, including robustness evaluation (RQ1), feedback-driven refinement and error reduction (RQ2), and verification-based reliability assessment (RQ3).

We evaluate adv-CoT on 12 datasets spanning diverse reasoning types, including commonsense, factual, symbolic, and arithmetic tasks. Results demonstrate that adv-CoT effectively optimizes prompts and improves model output accuracy. Furthermore, combining Self-Consistency (SC) [18] with adv-CoT yields strong performance across multiple datasets. The framework is also highly extensible and can integrate with other prompt-related methods for further improvement.

To provide a clear roadmap, the remainder of this paper is organized as follows. Section 2 reviews related work on prompt optimization and adversarial learning. Section 3 introduces the key components and terminology that underpin the adv-CoT framework. Section 4 presents the full methodology, including the adversarial learning setup, feedback mechanism, optimization algorithm, and verifier design. Section 5 describes the experimental setup, covering datasets, backbone models, baselines, evaluation metrics, and implementation details. Section 6 reports and discusses the main empirical results, including significance analysis, ablation studies, and cross-model generalization. Finally, Section 7 concludes the paper by summarizing the findings, outlining limitations, and highlighting directions for future research.

2. Related Work

2.1. Prompt Optimization for Chain-of-Thought Reasoning

CoT prompting has been shown to significantly improve reasoning performance in LLMs [9,10]. However, whether prompts are manually created or automatically generated, their effectiveness is still mainly assessed through computationally intensive downstream tasks, with costs increasing as the number of candidate prompts, sampled reasoning paths, and task instances grows [13,30,31]. Recent research has proposed more systematic optimization methods, including black box optimization [32] and Bayesian optimization combined with Hyperband [33]. These methods still provide limited theoretical guarantees for reasoning trajectories. Methods focusing on process-level evaluation, such as process supervision and PRM800K [34], offer promising alternatives to evaluation based solely on final task outcomes, but they remain in early development. Meanwhile, studies on robust prompt optimization [35] and uncertainty-aware evaluation [36] highlight the need for a unified framework that can simultaneously improve efficiency, robustness, and reasoning quality in CoT prompting.

2.2. Adversarial Training

Generative Adversarial Networks (GANs) were first introduced by Goodfellow et al. [25], establishing a minimax game framework in which the generator is incentivized to produce high-fidelity outputs. This adversarial paradigm has achieved remarkable success in tasks such as image generation [25,26], super-resolution [37], and domain adaptation [38,39]. While GANs have been widely adopted in the field of computer vision [40], applying them to discrete domains like text generation remains challenging due to the non-differentiability of sampling operations [41,42]. To overcome these difficulties, various enhancements [43,44] have been developed to stabilize the training process and reduce exposure bias.

Beyond stability, diversification and robustness have also become key objectives in adversarial text generation. To alleviate mode collapse and enhance output variety, recent work incorporates diversity-promoting objectives and entropy-based regularization [45]. Robustness remains challenging due to noisy rewards and discriminator drift; adversarial regularization [46] and robust discriminator designs [47] help improve reliability under unstable adversarial feedback. These advances shift adversarial text generation toward more diverse and dependable modeling.

When applying GANs to LLMs, direct parameter updates are often infeasible due to the high computational cost. In this context, leveraging adversarial learning for prompt optimization presents a more practical and efficient alternative. adv-ICL [29] is the first to leverage adversarial learning to optimize prompts for in-context learning. adv-ICL has already established a formal theoretical foundation for applying GAN-style minimax optimization to prompt refinement, demonstrating that adversarial learning is a valid and effective mechanism for optimizing prompts. While motivated by similar principles, adv-ICL and our method differ in several key aspects. Specifically, compared to adv-ICL, adv-CoT not only considers the outputs of the generator during training but also incorporates the discriminator’s loss to filter and refine generated prompts. This joint optimization enables both the generator and the discriminator to improve simultaneously, thereby maximizing the effectiveness of the training framework. Moreover, adv-ICL suffers from training instability due to the inherent variability in prompt candidates generated by LLMs, resulting in fluctuating loss values and increased training iterations, which ultimately raise computational costs. Conversely, adv-CoT addresses this issue by introducing a feedback mechanism during the discriminator phase: it identifies errors and provides revision suggestions to guide the generation of more effective prompt variants. This targeted guidance substantially reduces the number of required training iterations, improving efficiency while preserving performance.

3. Key Components and Terminology

To establish a clear conceptual foundation for our adversarial reasoning framework, we summarize the core modules involved in the optimization process. These definitions clarify the roles and interactions among components before presenting the full methodology.

Generator. The generator produces an initial reasoning chain and a candidate answer for a given input. It acts as the source of model-generated outputs that will later be examined by the discriminator. In the adversarial setting, the generator aims to produce answers that are indistinguishable from human-written responses.

Discriminator. The discriminator evaluates whether an answer is human-generated or produced by the generator. It plays the adversarial counterpart to the generator: while the generator attempts to craft outputs that appear human-authored, the discriminator attempts to correctly identify synthetic content. This adversarial tension drives the refinement loop.

Proposer. The proposer analyzes the generator’s output and identifies potential shortcomings in the reasoning process. It provides structured feedback by explaining possible failure points and suggesting targeted corrections. This feedback serves as a reflective signal used in subsequent refinement.

Modifier. The modifier incorporates the proposer’s feedback to revise and refine the reasoning chain. It performs targeted rewriting—strengthening logic, correcting errors, or restructuring reasoning—to improve the likelihood that the updated answer can pass the discriminator’s evaluation.

Verifier (trained discriminator). The verifier is the final, trained form of the discriminator, used exclusively at the answer-validation stage. Rather than guiding training, it functions as a judging module to determine whether the final reasoning chain yields a correct solution.

Adversarial learning. Adversarial learning refers to the competitive optimization dynamic between the generator and the discriminator. The generator aims to minimize the discriminator’s ability to detect generated content, while the discriminator attempts to maximize its classification accuracy, thereby improving reasoning quality through adversarial refinement.

Minimax game. This adversarial interaction can be formalized as a minimax game. The generator selects strategies that maximize its chance of fooling the discriminator, whereas the discriminator selects strategies that minimize the generator’s success. This mutually opposing objective structure provides a principled modeling perspective for the optimization process.

Self-Consistency. Self-Consistency is a decoding strategy that aggregates multiple sampled reasoning paths and selects the final answer based on agreement (e.g., majority voting). It increases reliability by reducing variance across individual reasoning samples.

4. Methods

4.1. Adversarial Learning

Adversarial learning serves as the core component of the adv-CoT framework, which consists of two main modules: a generator and a discriminator. Both modules are powered by an LLM and utilize CoT prompting. The generator produces answers based on the given instruction and CoT demonstrations, while the discriminator evaluates whether the input answer is correct or generated by the generator, also using the instruction and CoT demonstrations for guidance.

In addition, a modifier module refines the prompts based on the discriminator’s judgments and revision suggestions. It generates new instruction-example pairs, and the system selects the optimal pair based on the computed loss value to update both the generator and the discriminator. The detailed training procedure is illustrated in Figure 2.

Generator: The generator G is guided by a prompt ${\mathtt{Prompt}}^G$, which consists of an instruction ${\mathtt{Instruction}}^G$ and a set of input–output demonstrations ${\mathtt{Demonstration}}^G=(I_1^G,O_1^G,\dots ,I_k^G,O_k^G)$, where $I_i^G$ denotes an example input and $O_i^G$ is the corresponding output. The generator takes a user query as input and produces an answer as output.

Discriminator: The discriminator D is guided by a prompt ${\mathtt{Prompt}}^D$, which consists of an instruction ${\mathtt{Instruction}}^D$ and a set of input–output demonstrations ${\mathtt{Demonstration}}^D=(I_1^D,O_1^D,D_1^D,\dots ,I_k^D,O_k^D,D_k^D)$, where $I_i^D$ is an example input, $O_i^D$ is the corresponding output, and $D_i^D$ represents the discriminator’s decision and rationale for that example. D takes as input the user’s question along with the answer generated by the generator and outputs a binary decision indicating whether the answer is likely to be genuine or generated.

Modifier: The modifier M, driven by an LLM, is guided by a prompt ${\mathtt{Prompt}}^M$ that consists of a single instruction. This instruction includes two parts: ${\mathtt{Instruction}}_i^M$, which specifies how to revise the generator’s and discriminator’s instructions (${\mathtt{Instruction}}^G$ and ${\mathtt{Instruction}}^D$), and ${\mathtt{Instruction}}_e^M$, which guides the modification of the demonstrations (${\mathtt{Demonstration}}^G$ and ${\mathtt{Demonstration}}^D$). The modifier operates under a zero-shot prompting setting. Based on this instruction, the modifier generates multiple prompt variants ${\mathtt{Prompt}}_V^G$ and ${\mathtt{Prompt}}_V^D$ for the generator and discriminator, respectively. These variants are then used to recompute the loss function, enabling iterative updates to both the generator and discriminator.

4.2. Feedback

In adv-ICL, the discriminator merely identifies correctness, offering limited actionable feedback for subsequent optimization. This limitation becomes more significant in the context of LLMs, where the parameters updated by the modifier M are expressed as explicit textual prompts. Due to the inherent randomness in LLM outputs, the direction of modification becomes uncertain, leading to instability in the loss function and increased computational cost. To address this issue, adv-CoT records the discriminator’s decisions, collecting input–output pairs that the generator fails to handle, as well as those that the discriminator misclassifies or cannot confidently judge. These challenging cases are then passed to the Proposer, an LLM-driven module that generates targeted revision suggestions. This process offers explicit guidance for the modifier, effectively directing the prompt updates and improving training efficiency.

4.3. Adversarial Learning Algorithm

Algorithm 1 provides a detailed description of our adv-CoT. Inspired by GANs and adv-ICL, the training objective loss function of adv-CoT is defined as follows:

$$\mathcal{J}(D,G)={\mathbb{E}}_{x,y\sim p_{data}}log\left(D(x,y)\right)+{\mathbb{E}}_{x\sim p_{data}}log\left(1-D(x,G(x\left)\right)\right),$$

(1)

where $p_{\mathrm{data}}$ denote the real data distribution, and $\mathbb{E}$ represent the expectation with respect to the indicated distribution. The discriminator D outputs one of two options: “(A) Correct ground truth” or “(B) Generated output.” The confidence scores associated with these discriminator outputs are treated as expectation values during iterative training. For the discriminator D, the objective is to maximize the loss (considering that confidence scores are negative) to better distinguish generated answers from real ones. Conversely, the generator G aims to minimize the loss to avoid being detected as incorrect by the discriminator. Thus, the entire training process can be formulated as a minimax optimization problem:

$$\underset{G}{min}\underset{D}{max}\mathcal{J}(D,G).$$

(2)

Algorithm 1 Adversarial Chain-of-Thought Optimization

1: Input: Generator G, Discriminator D, Proposer P, Modifier M   2: Input: Prompts $Promp{t}^G$, $Promp{t}^D$, $Promp{t}^P$, $Promp{t}^M=Instructio{n}_i^M/Instructio{n}_d^M$   3: Input: Number of iterations $Nu{m}_I$, demonstrations $Nu{m}_D$, max candidates $Nu{m}_C$   4: Input: Demonstration set S   5: for  $i=1$  to  $Nu{m}_I$  do   6:        Sample $Nu{m}_D$ examples from S and compute initial loss J   7:        for all target ∈ {instruction, demonstration} do   8:              P generates $Feedbac{k}^P$ from $Promp{t}^P$ and D’s outputs   9:              for $j=1$ to $Nu{m}_C$ do 10:                    M generates new candidate $C_{new}$ using $Promp{t}^M$, $Feedbac{k}^P$, and J 11:                    Compute loss $J_{new}$ for $C_{new}$ 12:                    if $J_{new}>J$ then 13:                         Update candidate and loss: $C\leftarrow C_{new}$, $J\leftarrow J_{new}$ 14:                         break 15:                    end if 16:               end for 17:         end for 18:         // Similarly optimize $Promp{t}^G$ for G to minimize J 19:         … 20: end for 21: Output: Optimized prompts $Promp{t}^G$, $Promp{t}^D$

The final outcome of the training yields a new set of prompts, ${\mathtt{Prompt}}_{\mathrm{new}}^G$ and ${\mathtt{Prompt}}_{\mathrm{new}}^D$, corresponding to an updated generator $G_{\mathrm{new}}$ and discriminator $D_{\mathrm{new}}$.

4.4. Verifier

The Verifier V refers to the trained discriminator $D_{\mathrm{new}}$. Given a user query, the updated generator $G_{\mathrm{new}}$ produces an answer, which is then evaluated by the verifier. The verifier performs r rounds of evaluation, and the answer with the highest confidence score (i.e., the one associated with the greatest loss value) is selected as the final output. While prior prompting methods have explored ways to verify LLM-generated outputs, most rely on the LLMs to make simple judgments without a clear evaluation criterion or standard. In contrast, during adversarial training, both the generator and discriminator evolve together. This co-training allows the generator to produce more accurate outputs and the discriminator to serve as a reliable verifier. Traditional applications of GANs often focus solely on the generator after training, overlooking the discriminator. However, in the adv-CoT framework, the discriminator offers explicit verification signals, effectively enhancing the reliability and accuracy of the final answers. A more detailed illustration can be found in Figure 3.

5. Experimental Setup

5.1. Datasets

Following previous work [48], we selected three types of datasets in our experiments. The details are as follows: (1) Commonsense & factual reasoning. This category includes CommonsenseQA (CSQA) [49], StrategyQA [50], OpenBookQA [51], the AI2 Reasoning Challenge (ARC-c) [52], a sports understanding task from the BIG-Bench benchmark [53], and BoolQ [54]. These datasets are used to evaluate performance on commonsense and factual reasoning capabilities. (2) Symbolic reasoning. We evaluate two symbolic reasoning tasks: Last Letter Concatenation and Coin Flip [9]. (3) Arithmetic reasoning. This category includes grade-school math problems from GSM8K [55], the challenging math word problem dataset SVAMP [56], as well as AQuA [57] and MultiArith [58], which are all used for assessing mathematical problem-solving abilities. The detailed descriptions of the datasets can be found in Appendix A.

5.2. Backbone Models

We evaluated both open-source and closed-source models across a broad range of datasets. For the open-source model, we selected Llama-3-8B-Instruct [59]. Regarding closed-source models, we employed GPT-3.5-turbo (specifically the gpt-3.5-turbo-0125 version) and GPT-4o-mini (specifically the gpt-4o-mini-2024-07-18 version). Additionally, in cross-model experiments, we included deepseek-chat, which operates in the non-thinking mode of DeepSeek-V3.2-Exp. To ensure the reproducibility of our experiments, we fixed the random seed to 42. During training, prompt optimization relies on the generator producing sufficiently diverse responses. During testing, the generator is also required to generate multiple distinct answers; therefore, no fixed random seed is imposed on the generator. During the experiments, the generator, discriminator, proposer, and modifier modules were all powered by the same backbone model, unless otherwise specified.

5.3. Baselines

We begin our experiments with CoT [9] prompting and enhance it with Self-Consistency [18], using 10 sampled reasoning paths per input. To comprehensively validate the effectiveness of our proposed framework, we introduce two additional baselines: (1) AutoCoT [13], which acquires diverse questions via clustering-based sampling and automatically generates reasoning chains to construct demonstrations using Zero-Shot-CoT, eliminating the need for manual design of task-specific demonstrations and significantly reducing human effort; (2) adv-ICL [29], a representative method of adversarial in-context learning, used to compare the performance differences in adversarial optimization ideas.

5.4. Evaluation Metrics

In all experiments, we report accuracy as the evaluation metric. All datasets used in this work are single-label reasoning tasks—either multiple-choice (e.g., CSQA, ARC-c, AQuA), yes/no classification (e.g., BoolQ, StrategyQA), or exact numeric prediction (e.g., GSM8K, MultiArith). In these settings, accuracy directly measures whether the model outputs the correct answer, and it is also the predominant evaluation metric in prior CoT research. Other metrics such as precision, recall, or F1-score are less informative because each sample contains only one ground-truth label and no class-imbalance or multi-label structure is involved.

Formally, accuracy is defined as:

$$\mathrm{Accuracy}={\displaystyle \frac{{\sum }_{i=1}^{N}1({\widehat{y}}_i=y_i)}{N}},$$

(3)

where ${\widehat{y}}_i$ denotes the model prediction, $y_i$ is the ground-truth label, and N is the total number of instances.

We follow a unified evaluation pipeline for all models: given an input question, the model produces either a single CoT sample or 10 samples in the Self-Consistency setting. Answers are extracted via regex-based parsing and mapped to the official labels. The full evaluation script, including extraction, SC aggregation, and error-handling rules, is released in our repository.

5.5. Implementation Details

To encourage diverse outputs, we set the temperature to 0.6 for the generator, proposer, and modifier, while the discriminator operated with a temperature of 0 for deterministic judgment. Considering the training cost, we set the number of reasoning paths evaluated by the verifier to 3. All datasets are configured with 5 samples, except for Letter, which is set to 4. More details about the parameters can be found in Appendix B. During the experiments, the log-probabilities of the output tokens are used as a reference signal for computing the loss. For each dataset, we initialized the generator prompt $Promp{t}^G$ and discriminator prompt $Promp{t}^D$ as follows. The instruction for the generator $Instructio{n}^G$ was set to “Let’s think step by step” [10]. We adopted initial demonstrations for the generator based on prior work [48]. For the discriminator $Instructio{n}^D$, we extended the adv-ICL [29] discriminator prompt by incorporating an explicit reasoning requirement: “Judge whether the answer is the correct ground truth or a generated fake answer, and provide the reasoning process. You should make a choice from the following two options: (A) correct ground truth, (B) generated fake answer.” The initial demonstrations for the discriminator used input–output pairs from the generator as input, while the output was constructed by directly appending $Instructio{n}^D$ to the input under a zero-shot prompting setting using GPT-4, to elicit the model’s reasoning process. The detailed prompts can be found in Appendix C.

6. Results and Discussion

6.1. Results

Table 1. Accuracy (%) of GPT-3.5-turbo, GPT-4o-mini, and Llama-3-8B-Instruct across commonsense, symbolic, and arithmetic reasoning benchmarks. The red color indicates a decrease, while the green color indicates an increase. The arrows represent downward and upward trends, respectively.

Methods	Commonsense & Factual						Symbolic		Arithmetic
	CSQA	Strategy QA	OpenBook QA	ARC-c	Sports	BoolQ	Letter	Coin	GSM8K	SVAMP	AQuA	MultiArith
GPT-3.5-turbo
CoT	78.2	73.8	85.6	88.4	87.3	69.2	65.6	77.2	82.2	83.3	62.2	97.6
Auto-CoT	77.2	66.2	83.3	86.1	86.3	70.1	70.3	100	78.2	82.3	59.8	96.6
adv-ICL	77.8	66.6	86.4	87.2	86.3	73.1	74.6	96.8	80.4	84.1	60.2	98.5
adv-CoT	78.2 ± 0.0	73.3 ↓ 0.5	86.8 ↑ 0.4	89.5 ↑ 1.1	91.8 ↑ 4.5	73.5 ↑ 0.4	76.0 ↑ 1.4	98.4 ↓ 1.6	85.9 ↑ 3.7	85.3 ↑ 1.2	66.1 ↑ 3.9	99.1 ↑ 0.6
CoT+SC	79.4	74.5	86.0	88.3	89.4	67.0	66.6	99.2	87.5	86.8	65.3	98.6
GPT-4o-mini
CoT	84.9	77.4	94.1	94.3	92.3	77.1	86.8	100	93.4	93.6	77.1	98.1
Auto-CoT	84.3	67.8	94.6	95.3	56.2	77.3	89.0	99.8	90.1	93.8	79.1	98.6
adv-ICL	83.9	76.1	94.0	94.7	89.8	77.1	87.4	100	92.7	93.4	78.7	98.6
adv-CoT	84.8 ↓ 0.1	79.3 ↑ 1.9	94.8 ↑ 0.2	95.3 ± 0.0	93.1 ↑ 0.8	77.5 ↑ 0.2	91.0 ↑ 2.0	100 ± 0.0	93.7 ↑ 0.3	94.1 ↑ 0.3	79.9 ↑ 0.8	98.6 ± 0.0
CoT+SC	85.6	77.5	95.1	94.6	90.6	77.5	88.2	100	94.2	94.6	83.4	98.3
Llama-3-8B-Instruct
CoT	73.6	61.3	77.0	80.3	86.8	59.4	57.1	89.6	79.6	86.7	48.4	97.5
Auto-CoT	68.8	51.3	67.6	69.1	63.7	49.1	66.0	92.8	66.7	62.6	50.0	78.1
adv-ICL	73.5	40.6	77.0	76.3	85.9	60.0	61.1	82.8	80.5	86.8	38.5	97.8
adv-CoT	74.0 ↑ 0.4	62.3 ↑ 1.0	74.6 ↓ 2.4	81.4 ↑ 1.1	83.8 ↓ 3.0	62.5 ↑ 2.5	61.6 ↓ 4.4	89.0 ↓ 3.8	82.3 ↑ 1.8	86.9 ↑ 0.1	53.1 ↑ 3.1	97.6 ↓ 0.2
CoT+SC	77.2	66.0	80.4	85.4	91.4	61.3	59.0	96.8	85.6	90.4	57.8	98.0

reports the performance of different reasoning methods across commonsense, symbolic, and arithmetic datasets. Overall, adv-CoT provides consistent improvements on most benchmarks. Although several gains remain moderate, repeated-run statistics show that the variance of adv-CoT is generally small, indicating that the method is stable despite introducing adversarial perturbations. These empirical results primarily address RQ2, as they evaluate whether adversarial interactions between the generator and discriminator improve the stability, accuracy, and error resistance of CoT reasoning across diverse tasks.

For GPT-3.5-turbo, adv-CoT achieves notable improvements on Sports (+4.5), GSM8K (+3.7), AQuA (+3.9), SVAMP (+1.2), and BoolQ (+0.4). These results suggest that adversarially generated intermediate steps help correct shallow or shortcut reasoning behaviors, particularly in numerical and factual reasoning tasks. Moderate gains are also observed on OpenBookQA and ARC-c. However, performance remains nearly unchanged on StrategyQA and slightly decreases on Coin (−1.6), indicating that tasks with limited reasoning depth or strong lexical biases may not benefit from adversarial reasoning signals.

For GPT-4o-mini, adv-CoT again yields stable improvements, though with smaller margins overall. The largest gains appear on StrategyQA (+1.9), Letter (+2.0), SVAMP (+0.3), and AQuA (+0.8). Results on commonsense and arithmetic benchmarks remain largely unchanged, reflecting GPT-4o-mini’s already strong inherent reasoning capabilities. Notably, adv-CoT does not reduce performance on any arithmetic dataset and maintains perfect accuracy on Coin across all runs.

For Llama-3-8B-Instruct, adv-CoT leads to meaningful gains on CSQA, StrategyQA, ARC-c, BoolQ, GSM8K, SVAMP, and AQuA. The method is particularly beneficial for natural-language reasoning tasks requiring multi-step justification. Slight performance drops occur on OpenBookQA, Sports, Letter, and Coin, suggesting that smaller open-source models may be more sensitive to adversarial perturbations, especially in symbolic tasks.

Across datasets, the magnitude of improvement varies in ways that align with their underlying reasoning characteristics. Arithmetic and factual tasks exhibit the largest gains, consistent with the observation that their reasoning errors tend to follow structured and recurring patterns—such as omitted intermediate steps or unsupported factual links—that can be effectively suppressed under adversarial refinement. Symbolic reasoning tasks show smaller or less stable improvements, likely because their discrete step transitions make correctness harder to infer from textual surface forms, reducing the discriminator’s ability to provide consistent gradient signals. Slight decreases observed in Letter and certain commonsense benchmarks appear to be associated with tasks requiring flexible associative reasoning, where overly constrained refinement may occasionally limit beneficial variability.

Qualitative examination further indicates that adversarial optimization tends to produce more concise and coherent reasoning trajectories in tasks with well-defined solution structures, while tasks with inherently diverse reasoning strategies display a broader range of outcomes. These trends collectively suggest that the effectiveness of adv-CoT is closely related to the degree of structural regularity present in the target reasoning process.

Across all models, arithmetic datasets such as GSM8K, SVAMP, and MultiArith show improved or comparable results, demonstrating that the introduced adversarial reasoning steps do not destabilize numeric reasoning. CoT+SC is included only as a reference baseline, and details regarding Self-Consistency are discussed in Section 6.4.1.

6.2. Significance Analysis

All statistical analyses were conducted in Python (version 3.12.5). We used the scipy.stats package for paired t-tests, numpy for vectorized operations, and a custom bootstrap implementation based on 10,000 resampling iterations. Cohen’s d for dependent samples was computed following the standard formula using the mean and standard deviation of paired differences. When controlling for multiple comparisons, we applied the Benjamini–Hochberg FDR procedure implemented in statsmodels.

To examine whether the improvements in adv-CoT are reliable, we conducted paired statistical tests on the twelve benchmark datasets. For each model, we calculated the accuracy difference between adv-CoT and CoT on every dataset. These differences were treated as paired observations. We then applied three analyses: a paired t-test, a 10,000-sample bootstrap procedure, and Cohen’s d for dependent samples.

Table 2. Significance test results comparing adv-CoT against CoT across models.

Model	Mean Improvement (%)	Paired t-Test p	95% Bootstrap CI	Cohen’s d
GPT-3.5-turbo	4.44	0.0266	[1.76, 8.04]	0.739
GPT-4o-mini	1.08	0.0134	[0.47, 1.83]	0.849
Llama-3-8B	0.98	0.189	[−0.32, 2.27]	0.404

reports the results. For GPT-3.5-turbo, adv-CoT shows a significant gain of +4.44% (p = 0.0266; 95% CI = [1.76, 8.04]). The effect size is medium to large (d = 0.739). For GPT-4o-mini, the gain is smaller but still significant (+1.08%, p = 0.0134; 95% CI = [0.47, 1.83]). Its effect size is large (d = 0.849). For Llama-3-8B-Instruct, the improvement is modest (+0.98%) and not significant (p = 0.189; 95% CI = [−0.32, 2.27]). The effect size is small to medium (d = 0.404).

These results show a clear pattern. adv-CoT provides stable and meaningful improvements on high-capacity proprietary models. The gains on smaller open-source models are less consistent and vary more across datasets. The combined evidence from significance testing and effect sizes supports this conclusion.

Since each backbone model is evaluated over twelve datasets, the paired significance tests involve multiple comparisons. To control the increased risk of Type I errors, we applied the Benjamini–Hochberg False Discovery Rate (FDR) correction to all p-values. The corrected values do not change any of our main conclusions: the improvements remain significant for GPT-3.5-turbo and GPT-4o-mini, while the results for Llama-3-8B-Instruct remain non-significant.

6.3. Ablation Study

The ablation analyses provide additional evidence for RQ1 and RQ3, by disentangling how feedback, verification, and different prompt components jointly shape the optimization dynamics and the reliability of the resulting reasoning process.

6.3.1. Ablation Study on Feedback and Verification Modules

To assess the individual contributions of the feedback and verification modules in our framework, we conduct ablation experiments on five datasets—ARC-C, Sports, BoolQ, Letter, and AQuA—using gpt-4o-mini. We compare three configurations: the full adv-CoT framework, adv-CoT without the feedback module (w/o feedback), and adv-CoT without the verification module (w/o verification). For a controlled comparison, the full and w/o-verification settings share the same optimized prompts, allowing us to isolate the effect of the verification stage. The results are summarized in Figure 4.

Across the five datasets, the full adv-CoT achieves accuracies of 95.3%, 82.3%, 81.0%, 90.5%, and 79.5%. Removing feedback yields 95.8%, 78.6%, 80.0%, 84.8%, and 76.8%, whereas removing verification results in 94.1%, 74.0%, 79.5%, 84.3%, and 76.1%. With the exception of ARC-C—where omitting feedback produces a marginal improvement of only 0.5 percentage points—both ablations lead to consistent performance drops on all remaining datasets. The most substantial decreases appear when verification is removed on Sports (−8.3) and Letter (−6.2), and when feedback is removed on Letter (−5.7) and Sports (−3.7). These patterns indicate that the verification module plays a disproportionately strong role in stabilizing performance on high-variance tasks such as Sports and Letter, while the feedback module is essential for ensuring reliable improvements across different reasoning categories.

Training dynamics exhibit the same trend. The full adv-CoT shows lower final training loss (13.3, 10.3, 10.3, 10.0, 11.6) compared with the w/o-feedback variant (15.0, 11.9, 14.6, 16.0, 17.33), demonstrating faster convergence and more stable optimization. In contrast, removing feedback yields noticeably higher loss and slower optimization, suggesting that feedback provides more useful refinement signals. Verification, meanwhile, primarily improves the robustness of the final predictions rather than the speed of convergence.

These results highlight the complementary roles of the two modules: feedback enhances the efficiency and consistency of the optimization process, while verification ensures the reliability of the final reasoning outputs. The observed degradation in both accuracy and optimization stability when either module is removed demonstrates that both mechanisms are indispensable components of the adv-CoT framework. Overall, both mechanisms play complementary roles: feedback drives efficient learning, and verification ensures the robustness of the final reasoning results.

6.3.2. Ablation on Instruction–Demonstration Contributions

The optimized prompt used in adv-CoT consists of two key components: a task-level instruction that specifies the reasoning procedure, and a set of demonstrations that provide concrete exemplars of the desired chain-of-thought format. Although both components are jointly refined during the adversarial optimization process, their individual contributions to model performance remain unclear. To better understand their respective roles, we conduct an ablation study that isolates the effects of each component.

We evaluate three optimization configurations: (i) instruction-only, where the demonstrations are fixed and only the task directive is refined; (ii) demonstration-only, where the instruction remains unchanged while the exemplars are optimized; and (iii) joint optimization, corresponding to the full adv-CoT setting. Experiments were carried out using gpt-4o-mini on three representative benchmarks—AQuA (arithmetic reasoning), Letter (symbolic pattern reasoning), and ARC-c (commonsense abstraction). These datasets span distinct cognitive domains and thus provide complementary perspectives on the behavior of the two prompt components.

Figure 5 present the results. Optimizing only the instruction yields accuracies of 77.9% (AQuA), 90.4% (Letter), and 94.3% (ARC-c). While these gains indicate that task-level guidance plays a meaningful role in encouraging more structured reasoning, its improvement is consistently lower than that obtained by optimizing demonstrations. Refining only the exemplars leads to stronger performance on two datasets—79.1% on AQuA and 94.6% on ARC-c—suggesting that example-driven corrections may exert a more direct influence on the model’s reasoning format. However, on the Letter dataset, demonstration-only optimization results in a slight decrease (89.2%) compared with the instruction-only setting, implying that fine-grained symbolic tasks may be more sensitive to instruction clarity.

The joint optimization setting achieves the highest accuracy across all datasets—79.9%, 91.0%, and 95.3%—indicating that the two components are complementary rather than redundant. The interaction between refined instructions and optimized exemplars appears to stabilize reasoning trajectories and reduce the likelihood of error propagation. In particular, the improvements observed on the ARC-c dataset suggest that commonsense abstraction benefits from both explicit procedural guidance and carefully constructed demonstrations, reflecting a synergy between high-level directive refinement and example-driven reasoning corrections.

Overall, this ablation study shows that neither instructions nor demonstrations alone can account for the full performance of adv-CoT. The most substantial improvements arise from their coordinated optimization, underscoring the importance of treating the prompt as a multi-component structure whose elements interact to guide reasoning quality.

6.4. Discussion

6.4.1. Difference Between Verification and Self-Consistency

Although both the verifier mechanism and Self-Consistency (SC) involve generating multiple reasoning paths, their selection principles differ fundamentally. The verifier evaluates each reasoning chain using a learned confidence-based criterion and selects the answer with the highest predicted reliability. In contrast, SC adopts a majority-voting scheme, selecting the most frequently occurring answer across sampled reasoning paths. To better understand the distinct and complementary roles of these two mechanisms, we evaluate four configurations on four datasets (OpenBookQA, Coin, Letter, and AQuA): (1) optimized prompts only, (2) optimized prompts with verifier, (3) optimized prompts with SC, and (4) optimized prompts with both verifier and SC.

Figure 6 presents the empirical results. The optimized-prompt baseline achieves accuracies of 84.2%, 94.3%, 75.4%, and 64.5%, respectively. Incorporating the verifier yields improvements to 87.4%, 95.1%, 75.6%, and 70.0%. Although the gains are modest on Letter (0.2 percentage points), the improvements on OpenBookQA (+3.2) and AQuA (+5.5) indicate that confidence-based filtering is especially beneficial for tasks involving noisy or unstable intermediate reasoning. This aligns with the intended role of the verifier: eliminating low-confidence or logically inconsistent reasoning chains.

SC produces stronger improvements on three of the four datasets, reaching 86.0%, 98.2%, 78.4%, and 71.2%. The substantial gain on Coin (+3.9 percentage points over the verifier) suggests that SC more effectively mitigates stochastic variation by aggregating diverse reasoning trajectories, thereby reducing exposure to single-path failures. The robustness brought by majority voting appears particularly advantageous for tasks sensitive to sampling noise, such as symbolic reasoning in Letter and arithmetic reasoning in AQuA.

When combining both mechanisms, performance further improves to 86.8%, 99.2%, 80.0%, and 71.6%, achieving the highest accuracy on all datasets. This pattern indicates that verification and SC address different failure modes: verification suppresses low-quality reasoning paths by confidence calibration, while SC benefits from reasoning diversity and reduces variance across candidates. The complementary effects of these mechanisms suggest that combining confidence-based selection with diversity aggregation leads to more reliable and robust inference.

Overall, the performance degradation observed when removing the verification component reinforces its contribution to answer reliability, while the consistently stronger gains offered by SC highlight its importance for stabilizing inference under sampling variability. These findings provide a more nuanced understanding of the roles played by confidence calibration and diversity consolidation within the adv-CoT framework.

6.4.2. Cross-Model Generalization Analysis

To examine whether adversarial refinement overfits to a single model family, we conduct a cross-model generalization study using GPT-4o-mini and DeepSeek-V3.2-Exp. Each model is used both as the optimization (train) model and the evaluation (test) model, producing six train–test combinations. Experiments are performed on AQuA, Letter, and Sports.

The results in

Table 3. Cross-model generalization results of adv-CoT across GPT-4o-mini and DeepSeek-V3.2-Exp. Each cell shows accuracy on AQuA/Letter/Sports.

	Test Model	GPT-4o-mini	DeepSeek-V3.2-Exp
Train Model
None		77.1/86.8/92.3	88.1/96.3/95.3
GPT-4o-mini		79.9/88.6/93.1	89.3/96.8/94.5
DeepSeek-V3.2-Exp		80.7/86.2/93.0	89.7/96.4/96.2

reveal several nuanced trends. First, within-model evaluation shows clear and consistent gains: adv-CoT improves performance when GPT-4o-mini trains and evaluates on itself, and similarly when DeepSeek trains and evaluates on itself. These results verify that adversarial refinement is effective when the optimization and inference models share the same reasoning structure.

Second, cross-family transfer is feasible but task-dependent. Prompts optimized using DeepSeek transfer positively to GPT-4o-mini on AQuA and Sports, while showing a slight decrease on Letter. Conversely, GPT-optimized prompts transfer to DeepSeek with improvements on AQuA and Letter, but produce a small decline on Sports. These mixed outcomes indicate that part of the refined reasoning structure generalizes across model families, while another part remains tied to model-specific inductive biases.

Third, the magnitude and direction of transfer effects vary across tasks. Arithmetic reasoning (AQuA) benefits the most from cross-model refinement, whereas factual reasoning (Sports) is more sensitive to model-specific characteristics, leading to occasional performance degradation.

Overall, the cross-model analysis suggests that adv-CoT does not rely solely on a single model family; it learns partially transferable reasoning structures. At the same time, the variability observed across tasks and model pairs highlights the importance of future work on architecture-agnostic adversarial prompt optimization.

7. Conclusions

In this work, we introduced adv-CoT, an adversarial framework for evaluating and refining Chain-of-Thought prompts without modifying model parameters. Through generator–discriminator interactions combined with feedback and verification mechanisms, adv-CoT enables structured prompt revision and improves accuracy across a range of reasoning tasks. Our experiments on twelve datasets demonstrate that adv-CoT provides consistent yet task-dependent performance gains, particularly for numerical and factual reasoning. Significance analyses further confirm that these improvements are statistically reliable for high-capacity proprietary models, while the gains on smaller open-source models remain less stable.

Despite these promising results, several limitations must be acknowledged. First, the effectiveness of adv-CoT is inherently constrained by the underlying LLM’s reasoning capability. When applied to smaller or less reliable models, the feedback and refinement processes may be misled by incorrect intermediate reasoning, thereby reducing optimization quality. Second, the generality of adv-CoT remains limited. Our evaluation focuses on four categories of reasoning tasks and does not extend to more heterogeneous domains such as code generation, scientific problem-solving, long-context inference, or multimodal tasks. The cross-domain robustness of adversarial prompt refinement therefore warrants further investigation. Third, the framework depends on access to token-level log probabilities, which are required for discriminator scoring and verification. Many proprietary LLMs do not expose log-probability outputs, which restricts the applicability of adv-CoT in settings where such information is unavailable. Broader adoption of the framework may thus be constrained by API design and model transparency. Fourth, although feedback and verification help reduce variability, the iterative adversarial optimization introduces non-trivial computational overhead. The framework requires multiple rounds of model interaction during both training and validation, which may increase latency and cost—especially when relying on commercial APIs. Careful hyperparameter selection is required to balance optimization quality with computational efficiency.

Future work will explore broader task domains, more scalable feedback mechanisms, and improved control over prompt modification to achieve greater generality and robustness. We hope that the released implementation and prompts will facilitate deeper investigation into principled and transferable prompt optimization strategies.