ACR: Adaptive Confidence Re-Scoring for Reliable Answer Selection Among Multiple Candidates

Abstract

With the improved reasoning capabilities of large language models (LLMs), their applications have rapidly expanded across a wide range of tasks. In recent question answering tasks, performance gains have been achieved through Self-Consistency, where LLMs generate multiple reasoning paths and determine the final answer via majority voting. However, this approach can fail when the correct answer is generated but does not appear frequently enough to be selected, highlighting its vulnerability to inconsistent generations. To address this, we propose Adaptive Confidence Re-scoring (ACR)—a method that adaptively evaluates and re-scores candidate answers to select the most trustworthy one when LLMs fail to generate consistent reasoning. Experiments on arithmetic and logical reasoning benchmarks show that ACR maintains or improves answer accuracy while significantly reducing inference cost. Compared to existing verification methods such as FOBAR, ACR reduces the number of inference calls by up to 95%, while improving inference efficiency—measured as accuracy gain per inference call—by a factor of 2× to 17×, depending on the dataset and model.

1. Introduction

As the scale of language models (LMs) increases, there has been increasing interest in leveraging their capabilities to solve complex problems across diverse domains [1,2,3]. In particular, large language models (LLMs) have demonstrated strong reasoning abilities in question answering (QA) tasks through in-context learning (ICL), even without fine-tuning [4,5,6]. One prominent strategy for enhancing LLM reasoning is Chain-of-Thought (CoT) prompting [7], which encourages step-by-step reasoning before generating a final answer. This technique has been shown to substantially improve QA performance across a variety of tasks [8,9,10].

Self-Consistency (SC) [11] is typically used in combination with CoT prompting. It improves the consistency and reliability of final answers by selecting the most frequent answer among multiple reasoning paths generated by the LLMs. Since CoT encourages LLMs to perform multi-step reasoning, SC works more effectively when combined with CoT, enhancing both the accuracy and consistency of the final answers [11,12]. However, SC relies solely on frequency-based aggregation, which may fail to select the correct answer when it appears among the minority responses. For example, as shown in Figure 1, the LLM generates three unique candidate answers for a given question. Although the correct answer (189) is included among them, SC selects the most frequently generated response—an incorrect one (47.25)—as the final answer due to its majority voting strategy. This limitation is especially problematic when multiple answers occur with similar frequencies, leading to ambiguity and reduced confidence in the final prediction. To address this issue, recent methods such as Self-Verification (SV) [13] and FOBAR [14] have been proposed, which verify candidate answers before selecting the final answer. However, these approaches require performing separate verification for each candidate answer, resulting in substantial computational overhead.

We introduce Adaptive Confidence Re-scoring (ACR), a method that improves the reliability of final answers by collectively evaluating candidates generated by SC, rather than verifying each one individually. Unlike prior verification-based approaches, ACR formulates this process as a lightweight classification task over the candidate set, thereby minimizing computational overhead. ACR first computes an initial confidence score distribution based on the SC results and reformulates the evaluation process as a classification task. The LLM is then prompted to select the most plausible answer from the set of candidate answers, and the confidence scores are adjusted accordingly to compute the final score. The candidate with the highest final score is selected as the final answer. We evaluated the proposed method on five reasoning datasets—GSM8K [15], TabMWP [16], MATH500 [17], Word Sorting [18], and Reasoning about Colored Objects [18]—using five different language models: Qwen2.5-7B [19], Qwen3-14B [20], Mistral-Small-24B [21], Gemini-1.5-Flash-8B [22], and Gemini-2.5-Flash [3]. In all cases, ACR consistently outperformed SC across both arithmetic and logical reasoning tasks.

Our key contributions are as follows:

• We reduce computational overhead by selectively evaluating candidate answers based on the confidence distribution derived from SC.
• We reformulate the selection process as a classification task, enabling the model to recover correct answers even when they appear in the minority.
• We propose a collective evaluation strategy that enhances answer reliability without requiring separate verification for each candidate.

2. Related Work

2.1. Self-Consistency

Recent QA research has explored various strategies to improve answer accuracy by generating and aggregating multiple responses from LLMs. A notable early approach is Best-of-N (BoN) [23,24], which employs a reward model to score and select the most preferred output. More recently, Self-Consistency (SC) [11] has become a widely adopted alternative, selecting final answers based on consensus among reasoning paths rather than reward signals. SC generates multiple candidate answers by prompting the LLM to follow diverse reasoning paths and selects the final answer through majority voting. This method naturally filters out incorrect responses and tends to yield more reliable and accurate outputs [11,25,26].

To further enhance its performance, researchers have proposed several extensions that improve the robustness and flexibility of SC. Adaptive Consistency [27] dynamically adjusts the number of generated samples based on the problem instance, rather than using a fixed sample size, thereby maintaining efficiency without compromising performance. Soft Self-Consistency [28] replaces majority voting with continuous scoring based on token probabilities, improving the stability of answer selection. Additionally, paraphrasing-based methods [29,30] encourage diverse reasoning by constructing different input prompts for the same question.

While these approaches focus on improving SC—via sampling strategies, scoring mechanisms, or prompt diversification—our work instead aims to improve answer reliability after SC. Specifically, we propose a collective evaluation strategy that accounts for the possibility that minority answers may be correct, mitigating the risk of errors caused by frequency-based aggregation.

2.2. Verification

While SC selects answers based on consistency, it does not guarantee that the selected answer is always correct. To address this issue, recent approaches have incorporated an additional verification step to assess the validity of candidate answers. Self-Verification (SV) [13] performs backward reasoning (BR) for each candidate and selects the one with the highest verification score, regardless of the SC outcome. FOBAR (FOrward and BAckward Reasoning for verification) [14] combines SC and SV by computing both scores for each candidate and then selects the answer with the highest combined score. However, these methods require conducting BR individually for every candidate, resulting in substantial computational overhead. Moreover, verification is uniformly applied to all examples, which may lead to inefficiencies—especially when many cases are already confidently resolved by SC.

In contrast, our approach performs a collective evaluation over SC-generated candidates and applies it selectively, only to examples that require further scrutiny. This strategy improves both accuracy and efficiency by avoiding unnecessary evaluation.

3. Method

Our proposed method, Adaptive Confidence Re-scoring (ACR), consists of three main stages. Figure 1 illustrates the overall structure of ACR.

3.1. SC: Generating Initial Answers and Scores

In the first stage, we apply SC using few-shot CoT prompting (for GSM8K, TabMWP, Word, and Color) and zero-shot prompting (for MATH500 and AIME 2024) to generate M reasoning paths, yielding a set of M candidate answers $A={\left\{a_i\right\}}_{i=1}^M$. From these, we extract the set of unique answers $\mathcal{A}={\left\{{\widehat{a}}_c\right\}}_{c=1}^{\left|\mathcal{A}\right|}$ and compute the initial confidence score $Con{f}_1\left({\widehat{a}}_c\right)$ for each ${\widehat{a}}_c$ as follows:

$$Con{f}_1\left({\widehat{a}}_c\right)={\displaystyle \frac{Freq\left({\widehat{a}}_c\right)}{M}}$$

(1)

where $Freq\left({\widehat{a}}_c\right)$ denotes the number of times ${\widehat{a}}_c$ was generated. This score reflects the relative frequency of each candidate answer and serves as a measure of the model’s initial confidence.

3.2. SO: Selective and Collective Evaluation of Candidate Answers

ACR is designed to re-score confidence scores when an LLM generates inconsistent answers across multiple reasoning paths. However, performing verification on every data example—including those where the model consistently generates the same answer—can be inefficient. To address this, we introduce a selective evaluation step. Unlike prior verification-based methods discussed in Section 2.2, our approach employs a distinct strategy—termed Select Option (SO)—within the evaluation process. To enable selective evaluation, we utilize the initial confidence scores $Con{f}_1$ computed in Step 1 (SC). When the model’s answers are inconsistent, the resulting score distribution $Con{f}_1\left(\mathcal{A}\right)$ tends to be more uniform, exhibiting lower standard deviation.

As illustrated in Figure 2a, examples B and C show such uniform-like distributions, where no single candidate dominates—indicating that the model has generated inconsistent answers. In contrast, Figure 2b shows that example A has a more concentrated distribution with a higher standard deviation, reflecting a strong preference for a single answer and thus greater consistency. We use this standard deviation as a proxy for model uncertainty and trigger SO only when it falls below a predefined threshold $\delta$. This approach avoids unnecessary computation on examples where the model’s output is already consistent, thereby improving both efficiency and effectiveness.

For selected examples, we extract the top-K candidates from the initial set $\mathcal{A}$, denoted as ${\mathcal{A}}_K={\left\{{\widehat{a}}_k\right\}}_{k=1}^K$, and reformat them into a multiple-choice prompt, which is then fed back into the LLM. This reformulation transforms the open-ended QA task into a classification setting, where the model is asked to choose one answer from the given options. In SO, we sample N responses from the model: $O={\left\{o_j\right\}}_{j=1}^N$, where $o_j\in {\mathcal{A}}_K$. From the set of unique selections $\mathcal{O}={\left\{{\widehat{o}}_d\right\}}_{d=1}^{\left|\mathcal{O}\right|}$, we compute the second-stage confidence score $Con{f}_2$ as follows:

$$Con{f}_2\left({\widehat{o}}_d\right)={\displaystyle \frac{Freq\left({\widehat{o}}_d\right)}{N}},\hspace{1em}{\widehat{o}}_d\in {\mathcal{A}}_K$$

(2)

where $Freq\left({\widehat{o}}_d\right)$ denotes the number of times ${\widehat{o}}_d$ was selected.

3.3. ACR: Re-Scoring

Finally, we compute the final confidence score $Con{f}_{final}$ by applying a scoring function $G(Con{f}_1,Con{f}_2)$ to the confidence scores obtained from Step 1 and Step 2. The candidate answer ${\widehat{a}}_k$ with the highest $Con{f}_{final}$ is selected as the final prediction. In our implementation, the scoring function G is defined as follows:

$$G=Con{f}_1+Con{f}_2$$

(3)

This formulation simply aggregates the confidence scores derived from the two steps to produce the final score. We empirically find this scoring strategy to be the most effective, and compare it with alternative formulations in Section 6.1. ACR successfully flips the prediction by leveraging the evaluation stage—where the correct answer is selected by the majority—and re-scoring the candidates accordingly. This demonstrates ACR’s ability to recover correct answers in challenging scenarios where the LLM fails to generate consistent outputs.

4. Experimental Setup

4.1. Datasets

We evaluate our method on five datasets covering two types of reasoning tasks: arithmetic reasoning and logical reasoning.

4.1.1. Arithmetic Reasoning

• GSM8K  [15]: A dataset of grade-school-level math word problems. We use all 1319 examples from the test set.
• TabMWP [16]: From the TabMWP 1K test set, we exclude multiple-choice questions and evaluate on 751 examples whose answer types are either integer_number or decimal_number.
• MATH500 [17]: The MATH500 dataset comprises 500 challenging competition-level mathematics problems, uniformly sampled from the MATH dataset [31].

4.1.2. Logical Reasoning

• Reasoning about Colored Objects (Color) [18]: A subtask from BIG-Bench Hard benchmark, which requires reasoning over object–color relationships. Although the original format is multiple-choice, we remove the options and prompt the model in an open-ended format. We use all 250 examples.
• Word Sorting (Word) [18]: Another subtask from the BIG-Bench Hard that involves sorting a list of words in lexicographic order. The dataset contains 250 examples. Due to time constraints, we evaluate this task only with Gemini-1.5-Flash-8B.

4.2. Implementation Details

We evaluate ACR using five language models: Qwen2.5-7B (Qwen/Qwen2.5-7B-Instruct) [19], Qwen3-14B (Qwen/Qwen3-14B) [20], Mistral-Small-24B (mistralai/Mistral- Small-24B-Instruct-2501) [21], Gemini-1.5-Flash-8B (models/gemini-1.5-flash-8b) [22], and Gemini-2.5-Flash (models/gemini-2.5-flash) [3]. For MATH500, we conduct experiments using only Gemini-2.5-Flash, as the dataset consist of particularly challenging competition-level problems. Preliminary tests with smaller models yielded poor performance and unstable outputs, making them unsuitable for reliable evaluation on this benchmark. For all models except Qwen3-14B, we use the same generation configuration for both SC and SO: temperature T = 0.7, top-p = 1.0, and top-k = 50. For Qwen3-14B, we set temperature T = 0.7, top-p = 0.95, and top-k = 20, following its recommended decoding configuration, and set enable_thinking = False. Prompting strategies vary across datasets: we use few-shot chain-of-thought (CoT) prompting with 8-shot for GSM8K, 4-shot for TabMWP, and 3-shot for both Color and Word; for MATH500, we adopt zero-shot prompting. The maximum output length is set to 350 tokens for GSM8K and TabMWP, 1024 tokens for Word and Color, 1500 tokens for MATH500.

In Step 1 (SC), we generate $M=20$ responses per example. In Step 2 (SO), we set $N=5$ and consider the top-$K=4$ candidate answers based on their SC scores. The options presented in the SO input prompt are ordered according to their initial SC confidence scores, from highest to lowest. The choice of K is motivated by the typical number of distinct answers observed when the LLM exhibits inconsistency. Empirically, we found that even with $M=20$ samples, this number rarely exceeds four. Thus, setting $K=4$ provides sufficient coverage of plausible candidates while maintaining computational efficiency.

4.3. Prompt Design

We include representative prompt formats used in both the SC (Step 1) and SO (Step 2) stages to clarify how LLMs are guided during each stage of our framework. Figure 3 illustrates examples of the prompts used for GSM8K. While we adopt consistent prompt templates across datasets, the number of few-shot exemplars varies by task as described in the previous section.

4.4. Baselines

We compare our proposed method against the following baselines:

• SC [11]: Generates $M=20$ responses per example and selects the final answer via majority voting.
• SV [13]: Based on SC results, performs backward reasoning (BR) $N=5$ times for each candidate and selects the one with the highest verification score. We adopt the True–False Item Verification variant for consistency across datasets. For a fair comparison with ACR, BR is applied only to the top-K unique candidates from SC, using the same K value as in our method.
• FOBAR [14]: Extends SV by aggregating SC and BR scores using the weighting strategy proposed in the original FOBAR paper. We set the aggregation weight to $\alpha =0.5$.

5. Results

Table 1. Performance comparison across models and datasets. All values represent accuracy (%), with the best results shown in bold and second-best underlined. We compare baseline methods (SC, SV, and FOBAR) with our proposed method ACR, evaluated using four different scoring strategies. The $\delta$ value indicates the threshold used to decide whether ACR proceeds to Step 2 (Evaluation) and Step 3 (Re-scoring). Among the scoring variants, Sum is adopted as the default for comparison, due to its simplicity and performance. ACR consistently outperforms or closely matches the best-performing baseline across all settings. ACR variants are shaded.

Setting	Model	Dataset	Baseline			ACR
			SC	SV	FOBAR	Sum	Product	Fobar	SC-Boost	Setting
Few-shot	Qwen2.5-7B	GSM8K	91.96	91.81	93.25	92.34	91.96	92.49	92.72	$\delta =0.30$
		TabMWP	90.40	91.73	92.53	91.73	92.00	92.00	91.73	$\delta =0.30$
		Color	91.60	91.20	92.40	92.00	92.00	92.00	91.60	$\delta =0.30$
	Gemini-1.5-Flash-8B	GSM8K	86.88	87.26	87.64	88.40	88.17	88.48	88.25	$\delta =0.40$
		TabMWP	83.36	83.49	84.02	84.69	84.29	84.55	84.42	$\delta =0.40$
		Color	94.00	94.40	94.80	95.20	95.60	95.60	95.60	$\delta =0.20$
		Word	66.80	62.80	68.40	67.60	67.20	67.60	67.20	$\delta =0.05$
	Qwen3-14B	GSM8K	93.71	93.93	93.86	94.24	93.93	94.09	94.01	$\delta =0.40$
		TabMWP	95.87	97.33	97.07	97.07	97.20	97.20	97.07	$\delta =0.40$
		Color	98.80	98.00	98.80	98.80	98.80	98.80	98.80	$\delta =0.40$
	Mistral-Small-24B	GSM8K	94.84	94.77	94.92	94.91	94.91	94.91	95.29	$\delta =0.30$
		TabMWP	92.80	89.33	90.93	91.87	91.87	91.87	91.87	$\delta =0.30$
		Color	98.40	97.60	97.60	98.00	98.00	98.00	98.00	$\delta =0.30$
Zero-shot	Gemini-2.5-Flash	MATH500	91.60	91.00	91.20	92.00	91.40	91.40	91.60	$\delta =0.30$

presents a comparison between baseline methods and our proposed approach, ACR, across five datasets and five different LLMs. Among the four scoring strategies evaluated within ACR, we adopt Sum as the default. Details of all scoring variants are provided in Section 6.1.

Experimental results show that ACR consistently outperforms SC across all settings. While it does not always achieve the highest absolute accuracy, it performs comparably to FOBAR, which builds on SV-based scoring. Notably, there are cases where both SV and FOBAR underperform relative to SC, whereas ACR consistently maintains or improves performance. This trend holds not only for relatively easier datasets such as GSM8K and TabMWP, but also for a more challenging dataset like MATH500, where ACR achieves the best performance among all compared methods.

As shown in Figure 4, SV (and by extension FOBAR) performs separate verification for each candidate generated by SC, resulting in computational overhead proportional to the number of unique answers. In contrast, ACR employs a collective evaluation strategy that requires a fixed number of inference calls per example, independent of the number of candidates. Furthermore, ACR selectively triggers evaluation based on the standard deviation of the SC score distribution, often skipping additional inference altogether. This structural efficiency, coupled with robust performance, positions ACR as a practical and scalable alternative. A detailed analysis of computational cost is provided in Section 6.4.

6. Analysis

In this section, we analyze the key components of ACR and evaluate their impact on overall performance. We begin by comparing alternative scoring strategies used to combine the two confidence scores, $Con{f}_1$ and $Con{f}_2$, then examine the effectiveness of selective re-scoring, and finally assess the trade-off between accuracy and efficiency in comparison to FOBAR.

6.1. Final Confidence Scoring Strategies

To examine the robustness of ACR to different methods of integrating confidence scores, we evaluate four scoring functions for computing the final score $G(Con{f}_1,Con{f}_2)$ by combining $Con{f}_1$ from Step 1 and $Con{f}_2$ from Step 2:

(1)

Sum: A simple additive strategy that assigns equal weight to both scores. While intuitive and easy to implement, it may overemphasize noisy SO scores when $Con{f}_1$ is already high:

$$G_{Sum}=Con{f}_1+Con{f}_2$$

(4)

(2)

Product: A strictly multiplicative strategy that enforces agreement between SC and SO. However, it may excessively penalize candidates when either score is low:

$$G_{Product}=Con{f}_1\times Con{f}_2$$

(5)

(3)

Fobar: A softly weighted geometric mean, proposed by Jiang et al. [14] and controlled by a tunable parameter $\alpha$:

$$G_{Fobar}={Con{f}_1}^{\alpha }\times {Con{f}_2}^{1-\alpha }$$

(6)

Following the original implementation, we set $\alpha =0.5$ in our experiments.

(4)

SC-Boost: A hybrid strategy that augments $Con{f}_1$ with a multiplicative boost from $Con{f}_2$, thereby reflecting both the generation and evaluation stages:

$$G_{SC\_Boost}=Con{f}_1+Con{f}_1\times Con{f}_2$$

(7)

As shown in Table 1, all four strategies yield comparable performance across datasets, suggesting that ACR is robust to the choice of scoring function. Specifically, the average accuracy for each strategy is: Fobar (91.36%), Sum (91.35%), SC-Boost (91.30%), and Product (91.24%). Although Fobar achieves the highest average accuracy, we adopt Sum as the default scoring function in this work due to its simplicity, consistently strong performance, and lack of tunable hyperparameters.

6.2. Robustness to Option Order

In the SO stage, candidate answers obtained from SC are converted into a multiple-choice format, and the LLM is prompted to select one of the provided options. We adopt an ordering strategy in which candidates are sorted by their initial confidence scores ($Con{f}_1$) from Step 1, placing higher-confidence candidates earlier in the list. However, prior work [32] has shown that LLMs can exhibit inherent selection biases in multiple-choice tasks, making them sensitive to the ordering of options.

To assess the potential impact of this bias, we conducted a comparison on the GSM8K dataset using Mistral-Small-24B and Qwen2.5-7B, evaluating our $Con{f}_1$-based ordering against a random ordering. As shown in

Table 2. Accuracy (%) comparison across different answer selection methods. ACR ($Con{f}_1$) ranks candidate answers based on SC confidence, while ACR (Random) uses random order. All methods are evaluated with m = 20, n = 5, and K = 4.

Model	SC	SV	FOBAR	ACR (${\mathit{Conf}}_1$)	ACR (Random)
Qwen2.5-7B	91.96	91.81	93.25	92.79 ($\delta$ = 0.25)	92.87 ($\delta$ = 0.25)
Mistral-Small-24B	94.84	94.77	94.92	95.29 ($\delta$ = 0.25)	95.06 ($\delta$ = 0.25)

, the accuracy of Mistral dropped slightly from 95.29% ($Con{f}_1$) to 95.06% with random ordering, though the difference is minimal. For Qwen, the respective accuracies were 92.79% and 92.87%, indicating virtually no difference. These results suggest that our SO method is relatively robust to variations in option ordering.

Table 3. Comparison of accuracy (%) between SV and SO across datasets and models. While SO is applied selectively in ACR, here it is applied uniformly to all examples for a fair comparison with SV.

Dataset	Model	SV	SO
GSM8K	Qwen2.5-7B	91.81	89.83
	Gemini-1.5-Flash-8B	87.26	87.79
	Qwen3-14B	93.93	93.86
	Mistral-Small-24B	94.77	94.00
TabMWP	Qwen2.5-7B	91.73	87.60
	Gemini-1.5-Flash-8B	83.49	84.02
	Qwen3-14B	97.33	96.53
	Mistral-Small-24B	89.33	89.20
Color	Qwen2.5-7B	91.20	91.20
	Gemini-1.5-Flash-8B	94.40	90.00
	Qwen3-14B	98.00	98.00
	Mistral-Small-24B	97.60	97.20
MATH500	Gemini-2.5-Flash	91.00	91.80

compares the accuracy of SV and the SO stage in ACR. While ACR selectively applies SO based on the SC score distribution, for a fair comparison, we evaluate SO by uniformly applying it to all examples. As shown in Table 3, this often results in lower accuracy than SV. For instance, the accuracy gap reaches 4.40% for Gemini-1.5-Flash-8B on the Color dataset and 4.13% for Qwen2.5-7B on TabMWP. However, even in these cases, Table 1 shows that ACR still matches or even outperforms SV overall. This indicates that the performance of SO alone does not critically undermine ACR’s effectiveness. In particular, ACR does not rely solely on SO but incorporates a selective evaluation strategy that avoids unnecessary second-stage inference when the SC confidence is already high. Thus, the lower standalone performance of SO does not directly translate into a degradation in ACR’s overall performance. Nevertheless, as the performance of SO can still affect ACR in certain borderline cases, improving SO remains an important direction for future work. Enhancing SO’s reliability may lead to further gains in ACR’s accuracy without sacrificing its computational efficiency.

6.3. Effectiveness of Selective Re-Scoring

Figure 5 illustrates how the performance of ACR varies with different values of the threshold $\delta$. Across all three datasets, ACR maintains stable accuracy over a broad range of $\delta$ values, showing minimal sensitivity to the specific choice of threshold. This robustness indicates that the effectiveness of ACR does not critically depend on fine-tuning $\delta$, making it suitable for real-world applications where labeled data for validation may not be available. In particular, even on datasets such as TabMWP—where SO alone performs relatively poorly—ACR still exhibits consistent performance provided that $\delta$ is not set excessively high. These results highlight the practical benefit of ACR’s selective evaluation mechanism, which avoids unnecessary second-pass inference while preserving accuracy. Additionally, our analysis of performance across different $\delta$ values reveals that while the optimal threshold may vary depending on the model and dataset, thresholds in the range of 0.3 to 0.4 generally yield strong performance, as reflected in the $\delta$ values used for reporting ACR results in Table 1.

6.4. Efficiency and Effectiveness: ACR vs. FOBAR

To further evaluate the efficiency of ACR, we compare its best performance against SV, with both methods evaluated using $n=5$ generations. However, since FOBAR improves upon SV by applying its own BR-based scoring strategy, we compare ACR with FOBAR instead of SV to reflect the most competitive version. For ACR, we report the results using the $\delta$ value that yields the highest accuracy, as shown in Table 1.

$$\mathrm{Efficiency}={\displaystyle \frac{\mathrm{Accuracy}{\mathrm{Gain}}_{\mathrm{ACR}}/\mathrm{Inference}{\mathrm{Calls}}_{\mathrm{ACR}}}{\mathrm{Accuracy}{\mathrm{Gain}}_{\mathrm{FOBAR}}/\mathrm{Inference}{\mathrm{Calls}}_{\mathrm{FOBAR}}}}$$

(8)

As shown in

Table 4. Comparison of accuracy gain and inference cost relative to SC. Both FOBAR and ACR results are based on $n=5$ generations. For ACR, inference calls are computed using the $\delta$ value that yields the highest accuracy (see Table 1). “# of samples” refers to the number of data instances on which BR was performed in FOBAR, and the number of instances that proceeded to Step 2 and Step 3 in ACR, respectively. “Reduction” under “Inference Calls” indicates the percentage decrease in inference calls by ACR compared to FOBAR. “Accuracy Gain” denotes the increase in accuracy over SC. “Efficiency” is calculated using Equation (8), showing how much more efficient ACR is compared to FOBAR. We omit efficiency values when either FOBAR or ACR fails to outperform SC, as neither method is considered efficient in such cases.

Dataset	Model	# of Samples		Inference Calls			Accuracy Gain (%)		Efficiency
		FOBAR	ACR	FOBAR	ACR	Reduction (%)	FOBAR	ACR
GSM8K	Qwen2.5-7B	665	254	9615	1270	86.79	1.29	0.38	2.21
	Gemini-1.5-Falsh-8B	403	318	6685	1590	76.22	0.76	1.52	8.41
	Qwen3-14B	104	90	1345	450	66.54	0.15	0.53	10.52
	Mistral-Small	277	93	3605	465	87.10	0.08	0.07	7.04
TabMWP	Qwen2.5-7B	288	143	4020	715	82.21	2.13	1.33	3.52
	Gemini-1.5-Falsh-8B	214	178	3500	890	74.57	0.66	1.33	7.91
	Qwen3-14B	94	74	1310	370	71.76	1.20	1.20	3.53
	Mistral-Small	219	100	3155	500	84.15	−1.87	−0.93	-
Color	Qwen2.5-7B	87	31	1020	155	84.80	0.80	0.40	3.29
	Gemini-1.5-Falsh-8B	40	9	535	45	91.59	0.80	1.20	17.83
	Qwen3-14B	11	8	120	40	66.67	0.00	0.00	-
	Mistral-Small	44	4	460	20	95.65	−0.80	−0.40	-

, both FOBAR and ACR aim to enhance the SC baseline, but their computational costs differ significantly. FOBAR performs verification for each unique answer generated by SC, resulting in an inference cost proportional to the number of candidates. In contrast, ACR employs a selective, collective evaluation strategy based on the standard deviation of the SC score distribution. This allows ACR to either skip additional inference altogether or use a fixed number of calls, regardless of the number of candidates.

To quantify the trade-off between accuracy and cost, we compute both the accuracy gain per inference call and the overall efficiency ratio between ACR and FOBAR. As reported in Table 4, ACR consistently achieves comparable or better accuracy while requiring significantly fewer inference calls. For example, on GSM8K with Qwen2.5-7B, ACR reduces inference calls by nearly 87%, resulting in over 2× improvement in efficiency compared to FOBAR. On the Color dataset with Gemini-1.5-Flash-8B, ACR achieves an even greater efficiency gain of more than 17×, attributed to its selective evaluation and re-scoring mechanism.

In addition, Figure 6 illustrates how the number of inference calls varies with the threshold $\delta$. As $\delta$ increases, more data instances proceed to Step 2 and Step 3, leading to a gradual increase in computation. Nevertheless, ACR continues to require substantially fewer inference calls compared to SV or FOBAR across all threshold settings. These results underscore the advantage of ACR’s selective and collective evaluation strategy, which not only minimizes computational overhead but also preserves robust and stable performance.

7. Error Cases Analysis

In this section, we analyze instances where ACR fails to improve accuracy and categorize the main error types to better understand its limitations. We identify five recurring patterns:

• Failure to generate the correct answer: If the model fails to generate the correct answer in any of the SC reasoning paths, ACR cannot recover it during re-scoring. This represents a fundamental limitation of SC: if the correct answer is absent from the candidate set, no selection mechanism—ACR or otherwise—can succeed.

  Table 5. Proportion of incorrect predictions made by SC (Incorrect (%)), and within those, the percentage of cases where the correct answer was entirely missing from the generated candidate set (Gold Missing (%)). The latter represents cases fundamentally unrecoverable by rescoring methods like ACR.

  Dataset	Model	Incorrect (%)	Gold Missing (%)
  GSM8K	Qwen2.5-7B	8.04	31.13
  	Gemini-1.5-Flash-8B	13.12	41.04
  	Qwen3-14B	6.29	63.86
  	Mistral-Small	5.16	42.65
  TabMWP	Qwen2.5-7B	9.60	9.72
  	Gemini-1.5-Flash-8B	16.64	51.20
  	Qwen3-14B	4.13	29.03
  	Mistral-Small	7.20	29.63
  Color	Qwen2.5-7B	8.40	4.76
  	Gemini-1.5-Flash-8B	6.00	0.00
  	Qwen3-14B	1.20	33.33
  	Mistral-Small	1.60	0.00

  shows how often such unrecoverable cases occur.
• Consistently ambiguous cases: In some examples, both SC and SO assign similarly low confidence score gaps across multiple candidates, indicating persistent uncertainty throughout the pipeline. In such cases, ACR struggles to distinguish the correct answer from other plausible but incorrect ones.
• Correct answer favored in SO, but final prediction incorrect: Even when SO correctly assigns high confidence to the correct answer, the final prediction can still be wrong due to residual influence from SC.
• Overconfidence in an incorrect answer: SO occasionally assigns a disproportionately high confidence score to an incorrect candidate, even when SC’s distribution is relatively balanced. As discussed in Section 6.3, this illustrates how unreliable SO outputs can undermine ACR’s overall performance. These cases show that re-scoring may inject spurious certainty, leading ACR to reinforce an initially weak—but ultimately incorrect—prediction.
• Correct answer in SC, incorrect shift in SO: In these cases, SC initially assigns the highest confidence score to the correct answer, but SO shifts the score distribution toward an incorrect one, resulting in a wrong prediction. This highlights the risk of re-scoring introducing harmful bias—even when the SC result was already correct. As in the previous case, this underscores the importance of SO reliability in the overall performance of ACR.

  Table 6. Breakdown of predictions by SC and ACR across all dataset–model pairs. Proportions are reported as percentages of the total evaluation samples. “SC O & ACR O” indicates both methods predicted correctly; “SC O & ACR X” denotes degradation by ACR; “SC X & ACR O” indicates improvement by ACR; and “SC X & ACR X” refers to cases where both methods failed.

  Dataset	Model	SC O & ACR O	SC O & ACR X	SC X & ACR O	SC X & ACR X
  GSM8K	Qwen2.5-7B	90.14 %	1.82 %	2.20 %	5.84 %
  	Gemini-1.5-Flash-8B	85.60 %	1.28 %	2.81 %	10.31 %
  	Qwen3-14B	93.18 %	0.53 %	1.06 %	5.23 %
  	Mistral-Small	94.23 %	0.61 %	0.68 %	4.48 %
  TabMWP	Qwen2.5-7B	88.27 %	2.13 %	3.47 %	6.13 %
  	Gemini-1.5-Flash-8B	81.63 %	1.73 %	3.06 %	13.58 %
  	Qwen3-14B	95.20 %	0.67 %	1.87 %	2.26 %
  	Mistral-Small	91.07 %	1.73 %	0.80 %	6.40 %
  Color	Qwen2.5-7B	90.40 %	1.20 %	1.60 %	6.80 %
  	Gemini-1.5-Flash-8B	94.00 %	0.00 %	1.20 %	4.80 %
  	Qwen3-14B	98.80 %	0.00 %	0.00 %	1.20 %
  	Mistral-Small	98.00 %	0.40 %	0.00 %	1.60 %

  shows how often such Case 3–5 errors lead ACR to make incorrect predictions.

8. Conclusions

We introduced Adaptive Confidence Re-scoring (ACR), a lightweight and effective approach for improving answer selection in SC. While SC often generates the correct answer among its candidates, its reliance on majority voting can lead to the exclusion of accurate but minority predictions. ACR addresses this limitation by selectively evaluating candidate answers based on the confidence score distribution—without requiring exhaustive evaluation or incurring substantial additional cost. Experimental results across multiple datasets and models show that ACR consistently outperforms SC. While verification-based methods such as SV and FOBAR also improve upon SC, they require significantly more inference calls. In contrast, ACR achieves performance that is comparable to or better than SV/FOBAR, while requiring far fewer calls, demonstrating strong cost-effectiveness. Moreover, ACR remains robust across a range of threshold values, indicating that it does not require extensive hyperparameter tuning for practical deployment. These findings position ACR as a scalable and efficient solution for answer selection, providing a favorable trade-off between accuracy and computational cost.

9. Limitations

While ACR demonstrates strong performance and efficiency across multiple reasoning tasks, several limitations remain. First, ACR relies on the assumption that the correct answer is present among the candidate answers generated by SC. As discussed in Section 7, when the model fails to generate the correct answer in any reasoning path, ACR cannot recover it, since its re-scoring mechanism operates only over the existing candidates. Second, the effectiveness of ACR depends on the quality of the SO stage, which reformulates answer selection as a classification task. As shown in Section 6.3 and Section 7, SO can occasionally assign high confidence to incorrect answers or shift the final decision away from the correct one. These cases highlight the need for more effective prompting strategies within SO. In future work, we plan to explore more sophisticated prompt designs to improve the reliability of the evaluation stage. Third, although ACR reduces inference cost relative to SV and FOBAR, it still requires additional sampling beyond SC, particularly in the SO stage. In resource-constrained environments, even this reduced overhead may be non-trivial—especially in large-scale deployments. Finally, ACR currently uses a fixed scoring function (e.g., Sum), which may not be optimal across all datasets or model behaviors. Developing more adaptive or learnable scoring functions remains a promising direction for future research.