Weak-to-Strong Honesty Alignment via Group-Relative Policy Optimization

Abstract

Ensuring that Large Language Models align with human values of honesty is a critical challenge, particularly due to the scarcity of labeled data for distinguishing known versus unknown knowledge boundaries. We propose a weak-to-strong generalization framework utilizing Group Relative Policy Optimization (GRPO). Unlike standard supervised fine-tuning or prompt engineering, our framework trains a lightweight “honest head” to rank response candidates based on multifaceted honesty scores. Crucially, we employ GRPO to optimize this head, leveraging group-relative advantages and PPO-style clipping to robustly learn from noisy, relative honesty signals. The weak honest head then guides the self-labeling of unlabeled data to fine-tune strong LLMs. Experiments on PopQA, SQuAD, Non-AmbigQA, and a domain-specific military medical dataset demonstrate that our framework significantly outperforms strong baselines, including Direct Preference Optimization (DPO), in honesty alignment.

1. Introduction

In recent years, large language models (LLMs) have achieved significant progress in tasks like code generation [1,2], mathematical reasoning [3], and scientific research [4,5]. However, LLMs still frequently produce outputs that are factually inconsistent or lack grounding [6,7], undermining their reliability. Within the umbrella of Trustworthy AI [8,9], honestyis a key pillar: Models should provide accurate answers within their knowledge scope and explicitly acknowledge uncertainty beyond it [10,11]. Yet, achieving honesty remains challenging. Current alignment practices often prioritize helpfulness over honesty [12], largely due to the difficulty of creating model-specific alignment data that distinguishes known from unknown domains. Without such data, models risk either overconfidence (hallucination) or excessive refusal.

Existing efforts fall into two main categories: prompt engineering and fine-tuning. While prompting [13,14] yields only modest improvements, fine-tuning methods (SFT, DPO, PPO) [10,15] depend heavily on high-quality, diverse training data. Constructing such datasets imposes significant computational and human-resource demands, limiting scalability.

To address these challenges, we introduce GREAT (Group RelativE AlignmenT), a framework inspired by weak-to-strong generalization [16]. We train a lightweight “honest head” using Group Relative Policy Optimization (GRPO) [17] to guide the LLM. Conceptually, GREAT represents a fundamental departure from the concurrent WHAT(List) family [18]. While WHAT(List) employs supervised ranking (e.g., ListMLE) that forces the model to fit noisy pseudo-labels, GREAT leverages an RL formulation. Through PPO-style clipping and group-relative advantages, GREAT inherently tolerates label noise by limiting the magnitude of policy updates, thereby preventing overfitting to imperfect supervision.

Specifically, our proposal is built around three tightly coupled mechanisms that operate at different stages of training. First, we train a lightweight honest head on the limited labeled data available using GRPO, enabling the model to perform relative ranking of candidate responses by their honesty rather than absolute classification. Second, to address the scarcity of annotated data, we scale this supervision through a large-scale self-labeling process, where the honest head’s ranking scores are combined with the base LLM’s likelihood estimates to generate high-quality pseudo-labels for unlabeled samples. Finally, using this synthesized dataset, we adopt a weak-to-strong fine-tuning strategy to update the base LLM, thereby transferring the honest head’s specialized evaluative capability to the stronger model while retaining its generalization capacity. Together, these mechanisms yield substantial practical benefits. In high-stakes applications such as medical consultation, GREAT enables the model to autonomously recognize and refuse queries that exceed its competence, thereby reducing the risk of harmful or overconfident responses and significantly improving deployment reliability. In summary, this work makes the following contributions. We present a weak-supervision-based framework for improving honesty alignment in large language models, substantially reducing reliance on large-scale human annotation. We further introduce GREAT, a training paradigm that leverages group-relative rewards to achieve effective honesty alignment with minimal additional training cost and zero inference-time overhead. Extensive experiments demonstrate that GREAT consistently outperforms strong baselines in honesty-related benchmarks, while preserving the model’s intrinsic knowledge and overall performance.

2. Related Work

2.1. Honesty in LLMs

The concept of honesty in LLMs has recently gained significant research attention [10,11,19,20]. A truly honest LLM must provide accurate responses within its knowledge base while explicitly acknowledging uncertainty for queries beyond its scope [12]. Building on this principle, several studies have advanced the alignment of LLMs with honesty. For instance, Yang et al. [12] proposed evaluation frameworks and applied fine-tuning to enhance honesty, while Cheng et al. [10] utilized Direct Preference Optimization (DPO). Although these methods show promise, their efficacy is often constrained in low-resource scenarios with limited annotated data—a fundamental limitation addressed in this work.

2.2. Decoding in Language Models

Extensive research has focused on optimizing decoding strategies to improve generation outcomes. Early approaches employed auxiliary models to rerank translation outputs [21], while more recent studies explore human preference alignment during the decoding process [22,23]. Furthermore, some works utilize classifiers to guide generation for controllability [24] or hallucination reduction [25]. While the WHAT(List) series [18] formulates candidate selection as listwise Learning-to-Rank (LTR), our approach distinguishes itself by adopting GRPO with PPO-style clipping to ensure optimization stability. Crucially, existing methods often overlook honesty alignment and rarely frame candidate selection as a dedicated ranking task optimized through LTR objectives.

2.3. Weak-to-Strong Learning

The weak-to-strong generalization paradigm aims to enhance the capabilities of strong models using guidance from their weaker counterparts [16]. One research direction utilizes weak models to provide supervision via pseudo-labeling; for example, Burns et al. [16] established this framework for classification, while Yang et al. [26] extended it to complex reasoning. Alternatively, some methodologies integrate weak models directly into the generation pipeline [27,28]. For instance, Zhou et al. [28] employs log-probability differentials between tuned and untuned models as dynamic reward signals to steer decoding while maintaining efficiency.

2.4. Curriculum and Self-Reference Perspectives

Beyond the aforementioned approaches, recent curriculum-style strategies, such as LACT [29], suggest ordering training signals from easy to hard. In our framework, the honest head’s confidence scores serve as a natural proxy for task difficulty, enabling a curriculum that stabilizes early training phases. Furthermore, self-reference feedback [30] is complementary to our approach: the honest head can generate internal reference candidates to anchor groupwise comparisons during GRPO optimization.

3. Preliminary

This section presents preliminary knowledge regarding language model decoding and honesty measurement, which is helpful for understanding the subsequent methodology and for positioning honesty within trustworthy AI.

3.1. Decoding Process in LLM

In the decoding process of a large language model M, given an input token sequence q, the model can generate tokens at timestep t, where the probability distribution over the vocabulary is as follows:

$$P(y_t\mid y_{<t},q)=M(y_t\mid y_{<t},q),$$

where $y_{<t}$ represents the sequence of previously generated tokens. Afterwards, the model employs various sampling strategies to sample n candidates as $\{y_{1,t},\dots ,y_{n,t}\}$. The final decoding sequence $y^{*}$ is chosen based on an accumulated score, which is typically the log probability of the entire sequence:

$$Q(y^{*}\mid q)=\underset{y_{i,t}}{max}\sum _{t=1}^{T}logP(y_{i,t}\mid y_{i,<t},q).$$

(1)

Intuitively, $Q(·\mid q)$ is the sequence-level log-likelihood used as the base LM score; we will later ensemble it with the honest-head score when selecting pseudo-labels.

3.2. Measurement of Honesty

The measurement of honesty involves two steps: probing the model’s knowledge boundary and assessing the honesty of its responses.

3.2.1. Probing the Knowledge Boundary

This step aims to assess whether the question falls within the model’s knowledge domain. Let q be a question, and a be the reference answer to question q. We use LLM M to generate a set of responses $\mathcal{Y}=M(q,a)$. The honesty of a response $y_i\in \mathcal{Y}$ is evaluated by the correctness function $J(y,q,a)$ [12], which is defined as follows:

$$J(y,q,a)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}y\mathrm{is}\mathrm{the}\mathrm{correct}\mathrm{answer},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(2)

Here, $J(y,q,a)$ can be implemented through straightforward term matching or more sophisticated LLM judging [31].

To this end, we probe the model’s knowledge boundary through the lens of its generated responses, i.e., the proportion of correct responses in the set $\mathcal{Y}$ that exceeds a predefined threshold $\zeta$:

$$k(M,q)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\displaystyle \frac{{\sum }_{i=1}^{n}J(y_i,q,a)}{\left|\mathcal{Y}\right|}}>\zeta ,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(3)

Intuitively, $k(M,q)$ acts as an indicator of whether q lies within the model’s knowledge scope: it returns 1 when the fraction of correct candidates exceeds the threshold $\zeta$ (stricter with larger $\zeta$), and 0 otherwise.

3.2.2. Estimating Honesty Score

Quantifying honesty is challenging. We propose a heuristic composite score $v(y,q,a)$ that integrates correctness, refusal, knowledge boundary detection, and belief calibration. Instead of a complex piecewise definition, we formulate the score as a sum of a base correctness reward and a belief-adjusted term:

$$v(y,q,a)={\mathcal{S}}_{base}(y,q,a)+{\mathbb{I}}_{\mathrm{valid}}·{\lambda }_b·g\left(b\left(y\right|q)\right)$$

(4)

where ${\mathbb{I}}_{\mathrm{valid}}$ is an indicator function that activates the belief term only for valid (non-hallucinated) responses. The base score ${\mathcal{S}}_{base}$ is defined structurally:

$${\mathcal{S}}_{base}=\left\{\begin{array}{cc}3,\hfill & \mathrm{if}\mathrm{Correct}(\mathrm{Known} \& \mathrm{Answered}),\hfill \\ 2,\hfill & \mathrm{if}\mathrm{Prudent}\mathrm{Refusal}(\mathrm{Unknown} \& \mathrm{Refused}),\hfill \\ 1,\hfill & \mathrm{if}\mathrm{Over}\text{-}\mathrm{Conservative}(\mathrm{Known} \& \mathrm{Refused}),\hfill \\ 0,\hfill & \mathrm{if}\mathrm{Hallucination}(\mathrm{Unknown} \& \mathrm{Answered}).\hfill \end{array}\right.$$

(5)

This formulation clearly separates the discrete categorization of honesty behaviors from the continuous calibration signal provided by the model’s belief $b\left(y\right|q)$, offering a smoother optimization landscape for the ranker.

4. Weak-to-Strong Honest Generation

To orient readers, we first present the weak honest model (“honest head”) and input-layer selection, then describe Learning-To-Rank training and our group-relative alignment objective, and finally show how the head enables large-scale self-labeling followed by weak-to-strong fine-tuning. A concise comparison to pairwise/listwise training, including a discussion of listwise LTR vs GRPO, is provided in Section 4.2.

This section describes our method to generate pseudo-labels using a lightweight, easily trainable weak model, thereby reducing dependence on manual data annotation. Figure 1 provides an overview of our approach, GREAT. Our method comprises three phrases:

(1)

Weak Honest Model Training: We train a lightweight “honest head” model on limited labeled data within a Learning-To-Rank framework. This model is used to identify the most honest responses among those generated by LLMs via sampling.

(2)

Large-scale Self-labeling: The honest head model generates pseudo-labels for unlabeled responses, which provides reliable supervision for unseen instances.

(3)

Weak-to-strong Fine-tuning: Using pseudo-labeled data, we fine-tune a stronger, more robust model, enhancing its performance while minimizing manual annotation costs.

4.1. Weak Honest Model

To effectively guide the strong model during training, we first need a guide model to adapt to the target task. Through empirical observations, we find that although LLM-generated responses may exhibit dishonest behavior, the sampled candidate responses often contain some honest alternatives. This inspires us to adopt a re-ranking strategy to select honest responses from model outputs. We refer to the weak model as the honest head. Below, we detail the architecture of the honest head, the input selection strategy, and the learning-to-rank training approach.

4.1.1. Architecture Design

We instantiate the honest head as a 3-layer Multi-Layer Perceptron (MLP). This architectural choice is deliberate: strictly limiting the capacity of the honest head ensures it acts as a “weak” supervisor, forcing it to learn robust linear and non-linear features from the LLM’s hidden states without overfitting to the limited labeled data. This design aligns with linear probing methodologies [32,33], effectively extracting the model’s internal truthfulness representations while maintaining computational efficiency.

Formally, we estimate the honest score as:

$$s\left(y\right)=W_3·\sigma (W_2·\sigma (W_{1}h+b_1)+b_2)+b_3,$$

(6)

where $W_i\in {\mathbb{R}}^{d_i\times d_{i-1}}$ and $b_i\in {\mathbb{R}}^{d_i}$ denote weight matrices and bias terms, respectively $d_0$ input dimension, $d_1,d_2$ hidden dimensions, $d_3=1$ as the output dimension, and $\sigma$ represents the activation function. In particular, $h\in {\mathbb{R}}^{d_0}$ represents the hidden representation of the last token’s hidden state in response y, extracted from the selected transformer layer. The layer selection strategy is detailed in the following section.

4.1.2. Intermediate Layer Selection

Selecting appropriate hidden layers for honest head is crucial. We formulate the layer selection as an empirical optimization problem: for an N-layer transformer, we evaluate hidden states ${\left\{h_i\right\}}_{i=1}^N$ through grid search and select the layers that maximize honest score in Equation (6) on the validation set. We select the final token for the honesty score computation.

4.2. Group Relative Policy Optimization for Honest Head Training

Training the honest head to rank candidate responses by honesty presents unique challenges distinct from standard supervised classification. It is fundamentally a relative ranking problem, where the model must evaluate multiple candidates for each query and determine which is most honest given the context. Moreover, honesty labels are often noisy or incomplete, and different queries can yield candidate sets of highly variable sizes. These factors make traditional supervised objectives (e.g., cross-entropy) unstable and prone to overfitting.

Standard Learning-To-Rank (LTR) methods (e.g., pairwise or listwise losses) often struggle with high-variance gradients when labels are noisy or sparse. We instead adopt Group Relative Policy Optimization (GRPO). GRPO offers two distinct advantages for honesty alignment: (1) Variance Reduction: By normalizing rewards within a generated group (8), GRPO focuses on the relative honesty of candidates for a specific query, making the training robust to varying query difficulties. (2) Stability under Noise: The PPO-style clipping prevents the model from over-optimizing towards imperfect honesty proxies (pseudo-labels), a common failure mode in standard LTR.

4.2.1. Problem Formulation

We formulate the honesty alignment as a Group Relative Policy Optimization (GRPO) problem. Given a query q and a group of K candidate responses $\mathcal{Y}=\{y_1,\dots ,y_K\}$ generated by the base LLM, the honest head ${\pi }_{\theta }\left(y\right|q)$ acts as the policy. It assigns a scalar score $s_{\theta }\left(y\right)$ to each candidate, which is converted into a selection probability via a softmax distribution over the group:

$${\pi }_{\theta }(y_i\mid q,\mathcal{Y})={\displaystyle \frac{exp\left(s_{\theta }\left(y_i\right)\right)}{{\sum }_{j=1}^{K}exp\left(s_{\theta }\left(y_j\right)\right)}}.$$

(7)

To optimize the policy without an explicit critic network, we leverage the relative quality within each group. For each candidate $y_i$, we compute its raw honesty reward $r_i=v(y_i,q,a)$ according to the scoring mechanism defined in Equation (4). The group-relative advantage $A_i$ is then computed by standardizing the rewards within the group:

$$A_i={\displaystyle \frac{r_i-\frac{1}{K}{\sum }_{j=1}^K{r}_j}{\sigma \left(\mathcal{R}\right)+ϵ}},$$

(8)

where $\mathcal{R}=\{r_1,\dots ,r_K\}$ is the set of rewards for the group, and $\sigma (·)$ denotes the standard deviation. This standardization ensures that the optimization focuses on the comparative honesty of candidates, effectively mitigating the impact of absolute score variance.

4.2.2. Group-Relative Advantage Estimation

A critical innovation in our approach is the use of group-relative baselines to reduce variance. Absolute honesty scores can be misleading; a “low” score on a difficult question might still be the best possible answer in the group. Therefore, we compute the advantage $A_i$ for candidate $y_i$ relative to its peers:

$$A_i=\underset{\mathrm{Reward}\mathrm{Advantage}}{\underbrace{(r\left(y_i\right)-{\overline{r}}_{\mathcal{G}})}}+\gamma \underset{\mathrm{Policy}\mathrm{Self}\text{-}\mathrm{Correction}}{\underbrace{(s_{\theta }\left(y_i\right)-{\overline{s}}_{\mathcal{G}})}}$$

(9)

where ${\overline{r}}_{\mathcal{G}}$ and ${\overline{s}}_{\mathcal{G}}$ are the mean reward and mean score of the group $\mathcal{G}$, respectively. The first term rewards candidates that outperform the group average, effectively normalizing for query difficulty. The second term, scaled by $\gamma$, incorporates the policy’s current confidence, stabilizing the ranking updates as the model converges.

4.2.3. Optimization Objective

To ensure robust updates under noisy supervision, we adopt a PPO-style clipped objective. The total loss L combines the policy loss, a value function loss $L^{value}$ (for variance reduction), and an entropy regularization term $L^{entropy}$:

$$L=L^{policy}+\kappa L^{value}-\eta L^{entropy}$$

(10)

Specifically, the policy loss employs clipping to prevent destructive updates driven by imperfect pseudo-labels:

$$L^{policy}=-{\mathbb{E}}_t\left[min\left({\rho }_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}({\rho }_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t\right)\right]$$

(11)

where ${\rho }_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(y_t\right|q)}{{\pi }_{{\theta }_{old}}\left(y_t\right|q)}}$ is the probability ratio. This clipping mechanism is particularly vital for honesty alignment: it limits the incentive for the model to “game” the reward function, ensuring that policy updates remain conservative even when the heuristic honesty score $v(y,q,a)$ contains noise.

4.2.4. Training Procedure

The training iterates through three concise phases per batch:

• Generation & Scoring: For each query q, we sample a group of K outputs using Top-p sampling ($p=0.9$) with a temperature of $0.8$ and compute their honesty rewards $r\left(y\right|q)$.
• Relative Advantage Computation: We calculate group means (${\overline{r}}_{\mathcal{G}},{\overline{s}}_{\mathcal{G}}$) and derive the advantages $A_i$ via Equation (9), ensuring comparison is strictly local to the query.
• Update: The parameters $\theta$ are updated via gradient descent on L, optimizing the honest head to rank truthful answers higher while maintaining the stability provided by the clipped objective.

4.2.5. Comparison to Learning-to-Rank Methods

GRPO offers distinct advantages over traditional LTR approaches for honest head training. Pointwise methods evaluate candidates in isolation, missing the essential comparative aspect of ranking honest responses. Pairwise methods capture relative preferences but suffer from quadratic scaling with the size of the candidate set. Listwise approaches consider the full candidate set but lack variance control and stability mechanisms.

GRPO’s reinforcement learning formulation addresses these limitations through three key features: (1) explicit variance reduction via value baselines, producing more stable gradients; (2) inherent robustness to label noise through policy gradient optimization; and (3) stable updates via PPO clipping, preventing catastrophic policy changes during fine-tuning.

Empirically, GRPO outperforms listwise baselines (Section 5), particularly under noisy supervision or variable candidate quality. The actor-critic framework with advantage-based weighting provides an efficient and practical solution for selecting the most honest response from model-generated candidates.

4.3. Large Scale Self-Labeling

In this section, we elaborate on how to self-label with the honest head. Relying exclusively on the honest head may lead to suboptimal performance because it lacks the extensive semantic knowledge of the base model. Therefore, we combine honesty scores with the language model’s intrinsic likelihoods through ensemble decoding to generate pseudo-labels.

To generate K diverse response candidates, we employ Top-p sampling ($p=0.9$) with a temperature of $0.8$. This sampling-based approach ensures a wide variety of responses, which is essential for the subsequent ranking and group optimization phases.

Let ${\mathcal{Q}}_u$ be a set of unlabeled questions. For each question $q\in {\mathcal{Q}}_u$, language model M will generate K candidate sequences ${\left\{y_j^q\right\}}_{j=1}^K$ using beam search algorithm, we first obtain two scores for each candidate: the language model score $Q(y_j^q\mid q)$, defined in Equation (1) and the predicted honesty score $s\left(y_j^q\right)$ defined in Equation (6). We then combine these two terms as the final score via soft-attention normalization:

$$\begin{array}{cc}\hfill \widehat{Q}(y_j^q\mid q)& ={\displaystyle \frac{exp\left(Q(y_j^q\mid q)\right)}{{\sum }_{j=1}^{n}exp\left(Q(y_j^q\mid q)\right)}},\hfill \\ \hfill \widehat{s}\left(y_j^q\right)& ={\displaystyle \frac{exp\left(s\left(y_j^q\right)\right)}{{\sum }_{j=1}^{K}exp\left(s\left(y_j^q\right)\right)}},\hfill \\ \hfill z(y_j^q\mid q)& =(1-\omega )\widehat{Q}(y_j^q\mid q)+\omega \widehat{s}\left(y_j^q\right),\hfill \end{array}$$

(12)

where $\omega \in (0,1)$ is the honesty mixing ratio; it governs the contribution of honest head relative to the language model’s intrinsic likelihood. When $\omega =0$, this reduces to standard Best-of-N sampling. In practice, $\omega$ is tuned via a simple trial-and-error procedure on a validation set.

For each question $q\in {\mathcal{Q}}_u$, its most honest response is selected among the n generated responses according to the final score $z(y_j^q\mid q)$ as follows:

$$\begin{array}{cc}\hfill y_q^{*}& =arg\underset{y_j^q}{max}\left[z(y_j^i\mid q^i)\right].\hfill \end{array}$$

(13)

To this end, we can obtain the self-labeled dataset ${\mathcal{D}}_u={\{q,y_q^{★}\}}_{q\in {\mathcal{Q}}_u}$. To verify the reliability of this self-labeling process, we manually inspected a random sample of 100 generated pseudo-labels from the training set. Human verification achieved an accuracy of 86% in identifying the most honest response among candidates. While not perfect, this signal-to-noise ratio is sufficient for the weak-to-strong generalization paradigm, as subsequent GRPO fine-tuning is inherently robust to moderate levels of label noise.

4.4. Weak-to-Strong Generalization

Through honest head-guided decoding, we have already enhanced the model’s honesty. To further generalize the model’s ability, we fine-tune the model M using the self-labeling dataset ${\mathcal{D}}_u$ collected in Section 4.3.

It is important to clarify that the honest head is used exclusively during the self-labeling and training phases. Once training is complete, the honest head is discarded entirely. The deployed model is simply the fine-tuned base LLM, with no additional modules, parameters, or forward passes introduced at inference time. Therefore, the inference-time computational cost remains identical to that of the original backbone model, which is what we refer to as “near zero inference overhead.”

A large language model parameterized by $\theta$ is optimized by minimizing the negative log-likelihood loss over the self-labeled dataset:

$$\mathcal{L}\left(\theta \right)=-{\displaystyle \frac{1}{N}}\sum _{q\in {\mathcal{Q}}_u}logQ\left(y_q^{★}|q;\theta \right),$$

(14)

where $Q\left(y_q^{★}|q;\theta \right)$ denotes the probability of the pseudo-labeled output $y_q^{★}$ given input $x^i$ and model parameters $\theta$. The model parameters $\theta$ are updated to minimize this loss, resulting in the fine-tuned model $M^{*}$. This supervised fine-tuning process aligns the model’s output distribution with the high-quality pseudo-labels, enhancing both honesty and overall performance.

In summary, the pseudo-labels are generated by a weak model (i.e., small honest head), which are then used to enhance the honesty of a strong model (i.e., large language model). This weak-to-strong design endows our proposed framework with high generalization capabilities across diverse domains. In particular, GREAT achieves strong results even with scarce annotations due to two synergistic factors. On the one hand, the honest head uses a lightweight architecture (fewer parameters than the LLM) and a Learning-To-Rank loss. This allows it to effectively rerank the LLM’s outputs by honesty using minimal data. On the other hand, the LLM already possesses robust representations, enabling it to leverage pseudo-labels for further alignment. For unseen data, even imperfect pseudo-labels from the honest head allow the LLM to refine its latent capabilities.

5. Experiments

5.1. Experimental Setup

5.1.1. Baselines

To evaluate the effectiveness of our approach, we compared it against four baseline methods: (1) Prudent Prompt: This method provides the model with explicit instructions designed to encourage cautious reasoning and knowledge-aware responses. (2) In-Context Learning (ICL): The model is conditioned on four task-specific demonstrations to guide its reasoning. (3) Supervised Fine-Tuning (SFT): Due to computational constraints, we employed the LoRA fine-tuning approach [34], a parameter-efficient method that adapts the model via low-rank updates. (4) Direct Preference Optimization (DPO): This is a reinforcement learning-based optimization framework that aligns model outputs with human preferences. All prompts for these baselines are provided in Appendix C.

5.1.2. Datasets

We conducted experiments on three datasets: (1) PopQA [35] is a large-scale, open-domain question answering dataset consisting of entity-centric QA pairs. Each question is created by converting a knowledge tuple retrieved from Wikidata using a template. (2) SQuAD [36] is a reading comprehension dataset that contains questions posed by crowdworkers based on a set of Wikipedia articles. (3) Non-AmbigQA is a subset of the NQ-Open dataset [37], consisting of clear and unambiguous questions along with their corresponding answers. (4) MED is a sensitive dataset collected from open-source military medical equipment information. It comprises a series of questions and answers on the basic parameters and related descriptions of various types of military medical equipment from different countries. We deliberately chose this dataset as a test set for the model’s honesty-alignment capabilities, as such open-source, sensitive data are rarely used for model training, which can more comprehensively reflect the model’s true honesty-alignment capabilities.

Data Collection and Annotation: The data consists of questions and answers regarding parameters and descriptions of military medical equipment. All entries were manually curated and organized from open-source websites to ensure data quality.

Ethical Compliance: We strictly adhered to ethical guidelines during collection. As the dataset is derived entirely from the public domain (open Internet sources), it contains no classified or sensitive military intelligence. The final dataset consists of 280 Q&A pairs and serves solely as a test bed for out-of-distribution honesty generalization.

5.1.3. Experimental Details

In our experiments, we evaluated three open-source models: LLama3-8b-instruct [38], Gemma2-9b-instruct [39], and Mistral-7B-Instruct-v0.3. For brevity, we refer to these models as Llama, Gemma, and Mistral, respectively. We set $\alpha$ to 0.1. For supervised fine-tuning, we set the learning rate to 1 × 10⁻⁵ and used two epochs. During generation, models were configured with a temperature of 0.8 to encourage diversity.

All experiments were implemented using Python 3.10 and PyTorch 2.1.0. The models were trained and evaluated on an NVIDIA RTX 4090 GPU (NVIDIA, Santa Clara, CA, USA).

For supervised fine-tuning (SFT), we set the learning rate to 1 × 10⁻⁵ and used two epochs. For the Direct Preference Optimization (DPO) baseline, determining the Kullback–Leibler (KL) penalty coefficient is crucial for performance; we set $\beta$ to 0.1 following standard practices. The DPO training used a learning rate of 1 × 10⁻⁶ and a batch size of 64 to ensure stable convergence. A complete list of hyperparameters for all models and baselines is provided in Appendix B.2.

For Supervised Fine-Tuning (SFT) and GREAT, we employed LoRA with the following configuration to ensure parameter efficiency: rank $r=16$, scaling factor $\alpha =32$, and dropout rate $0.05$. The trainable parameters constitute approximately 0.1% of the total model weights.

The honest head module was trained for up to 40 epochs. We reserved 10% of the training data as a validation set to select the optimal layer’s hidden state as input to the honest head, based on validation performance. Both our proposed model (GREAT) and the DPO baseline underwent initial task-specific fine-tuning on labeled data to align their outputs with the task requirements. This fine-tuning used identical hyperparameters as SFT. Subsequently, the fine-tuned models generated outputs across the full training set, which were used to construct the honest head’s training data. The DPO baseline’s training data construction mirrored that of ${\mathrm{GREAT}}_{\mathrm{Pair}}$. For the first three open-source large datasets, we sampled 2000 examples from the training set (retaining answers) and removed answers from the remaining data to simulate unannotated questions. For datasets with provided documents, we discarded the documents and fed only the questions into the language model (LLM).

For the last self-built dataset MED, we directly tested the model trained by the GREAT framework on this dataset, which well reflects the honest performance of the model trained with honest heads in unknown vertical fields. Dataset statistics are detailed in Appendix B.

5.1.4. Evaluation Metrics

Following the methodology of Yang et al. [12], we evaluated model performance using three metrics: (1) Prudence Score: Measures the probability that a model expresses uncertainty when encountering unknown questions. (2) Over-Conservativeness Score (Over-Conserv.): Quantifies the probability that a model expresses uncertainty when responding to known questions. (3) Honesty Score: We report a consolidated honesty percentage defined as the arithmetic mean of Prudence and the complement of Over-Conservativeness, i.e., $\mathrm{Honesty}=\frac{1}{2}\left(\mathrm{Prudence}+(100-\mathrm{Over}\text{-}\mathrm{Consv}.)\right)$. This makes higher values consistently better and matches the up/down arrows used in the table.

To assess correctness, we strictly employ DeepSeek-V3 for fact-checking rather than subjective scoring. Since our datasets (PopQA, SQuAD) consist of factoid questions with determinate ground truth answers, the judge model primarily performs fuzzy string matching and semantic equivalence checks. Recent benchmarks demonstrate that state-of-the-art LLMs achieve near-human agreement (Kappa > 0.9) on such objective evaluation tasks [31].

5.2. Main Experimental Results

Table 1. Performance comparisons on PopQA, SQuAD, and Non-AmbigQA datasets. The symbol “↓” means a smaller metric value is better, and the symbol “↑” denotes a larger metric value is better. The highest honest score is highlighted in bold. Statistical significance compared to the strongest baseline is marked with * ($p<0.05$).

		PopQA			SQuAD			Non-AmbigQA			MED
		Prudence ↑	Over-Consv. ↓	Honesty ↑	Prudence ↑	Over-Consv. ↓	Honesty ↑	Prudence ↑	Over-Consv. ↓	Honesty ↑	Honesty ↑
Llama	Prompt	3.73	8.13	47.80	5.59	2.74	51.43	16.78	4.85	55.97	35.71
	ICL	5.40	4.44	50.48	7.25	2.59	52.33	13.11	5.41	53.85	33.65
	SFT	47.63	13.01	67.31	29.09	13.47	57.81	32.15	13.21	59.47	30.89
	DPO	49.31	11.35	68.98	34.70	15.88	59.41	29.10	11.52	58.79	38.84
	WHATPair	59.31	12.63	73.34	33.14	10.56	61.29	40.09	15.86	62.12	52.34
	WHATList	60.23	15.25	72.49	33.78	9.10	62.34	39.69	13.91	62.89	52.92
	GREAT	67.60 *	22.87	72.37	36.11 *	11.25	62.43 *	41.52 *	14.28	63.62 *	53.32 *
Gemma	Prompt	24.89	9.87	57.51	13.69	5.37	54.16	16.21	6.71	54.75	23.45
	ICL	22.48	10.04	56.22	15.49	6.81	54.34	19.31	5.89	56.71	34.09
	SFT	65.32	19.94	72.69	38.10	14.58	61.76	35.81	10.37	62.72	30.79
	DPO	67.23	23.29	71.97	39.81	15.57	62.12	34.92	5.14	64.89	40.69
	WHATPair	67.31	10.63	78.34	45.07	11.62	66.72	43.33	11.27	66.03	41.07
	WHATList	69.31	10.95	79.18	46.09	10.31	67.89	45.10	10.68	67.21	40.90
	GREAT	70.88 *	21.05	74.92	47.23 *	11.12	68.06 *	46.07 *	11.89	67.89 *	42.44 *
Mistral	Prompt	4.15	4.41	49.87	7.02	6.08	50.47	8.63	8.35	50.14	32.09
	ICL	7.61	5.68	50.96	10.89	8.55	51.17	8.03	7.17	50.43	20.63
	SFT	56.31	11.69	72.31	30.03	10.29	59.87	32.11	14.33	58.89	38.88
	DPO	55.32	9.08	73.12	36.75	14.71	61.02	36.70	18.08	59.31	31.53
	WHATPair	68.74	13.04	77.85	40.06	11.30	64.38	42.12	13.69	64.21	45.39
	WHATList	69.19	12.57	78.31	39.19	9.15	65.02	41.99	14.37	63.81	40.01
	GREAT	72.05 *	11.23	80.41 *	39.98	11.02	64.48	43.82 *	15.02	64.40 *	41.93

presents the experimental results across three datasets. From the table, we have the following key observations:

(1) Our method mitigates the challenge of scarce labeled data and achieves state-of-the-art (SOTA) performance. While fine-tuning improves honesty scores, it underperforms in low-data regimes. For example, on the SQuAD dataset, Gemma achieves a 14% honesty score gain via supervised fine-tuning (SFT) compared to baseline prompt engineering. Our approach further boosts performance by 9.9%.

(2) Prompt engineering demonstrates limited effectiveness in aligning with “Honesty”. Despite efforts to enhance honesty via prompt-based or in-context learning (ICL) methods, models exhibit low prudence and over-conservativeness scores, indicating a reluctance to express uncertainty.

(3) GREAT outperforms listwise methods for training the honest head.

The GREAT surpasses listwise comparison in 6 of 9 experimental settings (3 models × 3 datasets). This advantage stems from GREAT’s direct optimization of full output rankings, which better aligns with task objectives than pairwise local comparisons.

(4) GREAT demonstrates superior comprehensive honest performance on small-scale datasets in vertical fields compared to other methods.

(5) Practical Significance: Beyond statistical significance (p-values), what matters for alignment is whether performance gains yield more reliable real-world behavior. On PopQA, GREAT improves the Honesty Score by an absolute margin of 24.5% over the baseline Prompting strategy, meaning the model correctly handles roughly one in four previously mishandled queries—either by providing a correct answer or issuing a prudent refusal. This represents a substantive increase in practical reliability rather than a merely statistical distinction.

Across all datasets, GREAT produces not only higher honesty scores but also a more favorable balance between prudence and helpfulness. In particular, the method reduces unnecessary refusals while still lowering hallucination rates, indicating that the model has learned a more calibrated notion of when to answer and when to abstain. These improvements are substantial in magnitude—e.g., the gains on PopQA and MED correspond to materially fewer incorrect confident answers—and therefore represent practical, task-level benefits beyond mere statistical significance.

This advantage highlights GREAT’s robustness and adaptability in specialized domains with limited data, underscoring its potential for practical applications in niche scenarios where data availability is constrained.

5.3. Influence of Honesty Ratio

We examine the impact of the honesty ratio hyperparameter $\beta$. Figure 2 presents the performance of Llama on PopQA. To better understand the optimality of $\beta \approx 0.4$, we conducted an additional analysis on a small held-out validation subset (200 samples). We observed that the honest head tends to be overly conservative on ambiguous entities, whereas the base LLM (Best-of-N) can be overly confident. The value $\beta =0.4$ effectively minimizes the KL-divergence between the ensemble distribution and the ground-truth honesty distribution on this subset. Conceptually, this value represents a coherent trade-off where the honest head acts as a regularizer to the LLM’s intrinsic semantic probability, filtering out high-likelihood but factually incorrect hallucinations without causing excessive refusal.

5.4. Statistical Significance and Sensitivity Analyses

We assess the robustness of the observed improvements through statistical testing and sensitivity analyses.

For each dataset/model, we compute paired t-tests over per-example honesty scores comparing GREAT to the strongest baseline. Across settings, we observe $p<0.05$ in the majority of cases; where numerical gaps are small, significance still holds due to consistent per-example gains. Detailed p-values are listed in the

Table A4. Paired t-test p-values comparing GREAT with the strongest baseline.

Dataset	Model	Strongest Baseline	Test	p-Value
PopQA	Llama	WHATPair/DPO	paired t-test	0.003
PopQA	Gemma	WHATPair/DPO	paired t-test	0.011
PopQA	Mistral	WHATPair/DPO	paired t-test	0.007
SQuAD	Llama	WHATPair/DPO	paired t-test	0.028
SQuAD	Gemma	WHATPair/DPO	paired t-test	0.019
SQuAD	Mistral	WHATPair/DPO	paired t-test	0.033
Non-AmbigQA	Llama	WHATPair/DPO	paired t-test	0.041
Non-AmbigQA	Gemma	WHATPair/DPO	paired t-test	0.023
Non-AmbigQA	Mistral	WHATPair/DPO	paired t-test	0.036
MED	All	Best baseline per model	paired t-test	0.018

. We also sweep $\beta \in \{0,0.2,0.4,0.6,0.8,1.0\}$ and report the mean Honesty and Over-Conservativeness. Results indicate a robust plateau around $\beta \in [0.3,0.6]$, balancing refusal and correctness. We provide a recommended default of $\beta =0.4$. Moreover, a grid search over hidden layers shows mid-to-late transformer layers typically yield stronger honesty discrimination, whereas the final layer is not always optimal. We include a summary table in the Appendix A.1, highlighting the best-performing layers for each backbone.

5.5. Error Analysis and Failure Modes

To provide a balanced evaluation, we analyzed typical failure cases of GREAT. We observed two primary failure modes:

• Over-Refusal on Ambiguous Entities: For questions involving rare but existent entities (e.g., niche medical equipment variants), the honest head occasionally assigns low scores due to low internal confidence, causing the model to conservatively answer “I am not sure”, even when it potentially “knows” the answer.
• Hallucination under Strong Misleading Prompts: Although GREAT reduces hallucinations, it is not immune to adversarial prompts designed to force an answer. In 15% of the error cases, when the prompt explicitly forbade refusal, the model reverted to hallucination.

Future work will focus on calibrating the honest head to better distinguish between epistemic uncertainty and ambiguity.

6. Limitations

This paper improves model honesty within the weak-to-strong generalization framework. However, several limitations remain. First, due to computational resource constraints, our evaluation is limited to models in the 7B-9B parameter range (Llama-3-8B, Gemma-2-9B, Mistral-7B). Scalability to larger models (e.g., 70B+) remains an empirical question. Theoretically, consistent with the findings of weak-to-strong generalization [16], we hypothesize that larger models would benefit even more from our framework, as they possess stronger intrinsic representations of truthfulness that can be more effectively elicited by the honest head.

Second, regarding uncertainty in model responses, this study adopts a binary classification (certain/uncertain) rather than quantifying uncertainty, an approach that could be extended in future work. Furthermore, we do not extensively explore variants of GRPO’s loss functions. Finally, although we extend honesty to a multi-factor score (correctness, refusal, knowledge boundary k, belief b), the current design is a simplified template; improved belief calibration, threshold selection for $\alpha$, and more granular refusal types are left for future work.

7. Conclusions

In this study, we introduce GREAT, a method that enhances model honesty by enabling weak-to-strong generalization. Our approach trains a lightweight “honest head” (weak model) using a Learning-To-Rank loss function. This head reranks beam-search candidates from the model’s output, enabling self-labeling of unannotated data. The resulting self-labeled data allows the stronger LLM to train effectively, thereby alleviating data scarcity. Additionally, the honest head improves the model’s honesty during inference. Extensive experiments demonstrate that GREAT effectively mitigates labeled data scarcity and achieves state-of-the-art results in honesty alignment.

Finally, GREAT maintains the inference efficiency of the base model. Because the honest head is only used to produce training-time signals and is removed afterward, the final model architecture at deployment matches the original backbone exactly. As a result, our approach introduces no additional latency or memory footprint during inference.