Do Large Language Models Know When They Lack Knowledge?

Abstract

Although Large Language Models (LLMs) excel in language tasks, producing fluent and seemingly high-quality text, their outputs are essentially probabilistic predictions rather than verified facts, rendering reliability unguaranteed. This issue is particularly pronounced when models lack the required knowledge, which significantly increases the risk of fabrications and misleading content. Therefore, understanding whether LLMs know when they lack knowledge is of critical importance. This work systematically evaluates leading LLMs on their ability to recognize knowledge insufficiency and examines several training-free techniques to foster this metacognitive capability, referred to as “integrity” throughout this research. For rigorous evaluation, this study firstly develops a new Question-Answering (Q&A) dataset called Honesty. Specifically, events emerging after the model’s deployment are utilized to generate “unknown questions,” ensuring they fall outside LLMs’ knowledge boundaries, while “known questions” are drawn from existing Q&A datasets, together constituting the Honesty dataset. Subsequently, based on this dataset, systematic experiments are conducted using multiple representative LLMs (e.g., GPT-4o and DeepSeek-V3). The results reveal that semantic understanding and reasoning capabilities are the core factors influencing “integrity.” Furthermore, we find that well-crafted prompts markedly improve models’ integrity, and integrating them with probability- or consistency-based uncertainty evaluation methods yields even stronger performance. These findings highlight the considerable potential of LLMs to express uncertainty when they lack knowledge, and we hope these observations can lay the groundwork for developing more reliable models.

1. Introduction

Large Language Models [1,2,3,4,5] have recently demonstrated extraordinary capabilities, achieving unprecedented performance in various applications. However, during real-world deployments, the reliability of LLMs poses a substantial challenge, as LLMs often make factually incorrect generations, referred to as hallucination [6,7,8], primarily when the required knowledge is missing from their parametric knowledge [9]. Therefore, assessing LLMs’ ability to recognize their own knowledge limitations is vital for real-world applications.

As illustrated in Figure 1, unlike conventional hallucination-detection studies, our work focuses on a more fine-grained setting: Identifying situations in which LLMs lack the requisite knowledge to answer a question. In such cases, humans are often aware of their own knowledge limitations and can explicitly express this uncertainty. We aim to investigate whether LLMs possess a similar capability—that is, whether they can reliably detect knowledge insufficiency and signal it either explicitly in their responses or implicitly through probabilistic outputs or latent representations. (Explicitly expression of uncertainty refers to cases in which LLMs verbally convey ignorance or low confidence (e.g., “I’m not sure”), whereas implicit expression of uncertainty is reflected in low output probabilities or internal model representations.)

Previous studies [10,11] investigate and improve LLMs’ awareness of knowledge boundaries by constructing known/unknown Q&A benchmarks and aligning models to answer when knowledgeable but abstain with “I don’t know” when not, supported by tailored training and evaluation frameworks. Nevertheless, they all oversimplify the distinction between errors and knowledge limitations by approximating incorrectly answered questions as beyond the model’s knowledge boundaries. This approximation is clearly unreasonable, and fundamentally undermines the credibility of their evaluation results, as an incorrect answer may be due to poor prompts or insufficient reasoning ability of models [9,12].

Therefore, a prerequisite for conducting this study is the availability of a trustworthy dataset containing instances that fall beyond the knowledge boundaries of LLMs. Yet it’s a Herculean task to determine the knowledge boundaries of LLMs due to the limited transparency of most models with respect to the vast pretraining data [10]. Given that the internal knowledge of the models remains static, we use the latest events occurring after the launch of the LLMs to generate unknown questions in Honesty to ensure that they lie beyond the knowledge boundaries of the models (See Section 3 for details). Based on this, we conduct extensive experiments on Honesty, involving 8 LLMs including the Llama series [4,13,14], GPT-4o [3] and DeepSeek-V3 [15], to examine whether LLMs can exhibit “integrity.” Additionally, we systematically analyze the key capabilities of LLMs influencing their performance and evaluate the effectiveness of several existing training-free techniques in enhancing models’ “integrity.” (“Integrity” refers to the capability of LLMs to recognize the situation when they lack knowledge). Unlike prior related work [10,11], we specifically focus on methods that do not require modifying model parameters through training or fine-tuning, such as prompt-based approaches and techniques relying on output confidence scores. These training-free methods offer superior applicability and generalization across diverse models by eliminating dependence on model-specific fine-tuning. Consequently, they offer greater versatility and scalability in heterogeneous deployment scenarios, while avoiding the substantial computational overhead and risk of overfitting inherent in fine-tuning-based approaches.

We discover that when prompted to answer unknown questions directly, all LLMs tend to generate hallucinations rather than explicitly indicating a lack of knowledge, suggesting that even state-of-the-art LLMs remain unreliable. This further underscores the necessity of conducting related research to address these limitations and improve model reliability. Additionally, LLMs showcase “integrity” stimulated by instructions, with the top-performing model, GPT-4o, achieving an F1 score of 78%. Moreover, our experiments reveal that chain-of-thought (COT) prompting [16] can further augment the “integrity” of LLMs, enabling more reliable recognition of knowledge insufficiency. In contrast, In-Context Learning (ICL) prompting yields no comparable improvement. On the one hand, COT prompting can enhance LLMs’ reasoning abilities; on the other hand, it can explicitly guide LLMs to recognize situations in which knowledge is lacking. For instance, questions that exceed one’s own knowledge scope in the temporal dimension or involve unfamiliar concepts or events may require external knowledge. Additionally, we observe that “integrity” is primarily influenced by two key capabilities of LLMs: their capacity for semantic understanding—often reflected in their ability to follow instructions—and their reasoning capability. These findings indicate that LLMs possess considerable latent potential to explicitly express uncertainty when faced with knowledge gaps; however, this capability requires explicit prompts or clear instructions. In the absence of such guidance, models continue to exhibit a pronounced tendency to produce incorrect and potentially misleading responses.

We also leverage several state-of-the-art (SOTA) probability- or consistency-based uncertainty evaluation methods, including INSIDE [17], Length-normalized Entropy [18], and Lexical Similarity [19], to assess the ability of LLMs to implicitly express uncertainty when confronted with knowledge gaps. The experimental results show that LLMs exhibit higher uncertainty when answering unknown questions, indicating that the models possess a certain degree of implicit uncertainty expression capability. However, their performance is similar to that of instruction-based prompting and significantly inferior to that of COT prompting. Nevertheless, combining COT prompting with probability- or consistency-based methods further enhances the models’ “integrity.” We clarify that our experiments across multiple LLMs are intended to examine the consistency of integrity-related behaviors and the effectiveness of training-free interventions, rather than to compare or rank model performance across different generations.

The main contributions of this work are as follows:

• This study introduces Honesty, a high-quality dataset comprising a range of well-crafted questions beyond the knowledge boundaries of the employed LLMs, thereby enabling more reliable evaluation.
• Several widely used LLMs with different sizes, versions and abilities, are employed to investigate whether they can exhibit “integrity” and analyze the primary capabilities influencing their “integrity”.
• Our research systematically evaluated the effectiveness of various prompting strategies in enhancing models’ “integrity,” demonstrating the potential of LLMs to explicitly express uncertainty when they lack knowledge.
• We explored the efficacy of several probability- or consistency-based uncertainty evaluation methods in identifying unknown questions and found that their integration with prompt-based approaches further enhances the models’ performance, which suggests that integrating both explicit and implicit methods could be a promising approach.

2. Related Work

2.1. Metacognitive Knowledge of LLMs

Derived from cognitive psychology [20,21], metacognitive knowledge encompasses the competence to self-reflect and critically evaluate their cognitive processes [22]. Several studies [23,24,25,26,27] investigate whether LLMs can recognize questions that don’t have definitive answers in general. However, in real-world applications, the primary focus is on whether LLMs can accurately answer fact-based questions, rather than handling ambiguous or unanswerable ones. Additionally, it is essential for LLMs to accurately acknowledge their lack of knowledge when encountering questions outside their scope. Recent works [10,11,28,29] have explored fine-tuning LLMs to explicitly express their confidence in responses or to decline answering questions that they can’t answer correctly. Nevertheless, they have not conducted a comprehensive analysis of the factors influencing LLMs’ “integrity,” nor have they precisely defined which questions fall beyond the models’ knowledge boundaries.

2.2. Uncertainty Evaluation of LLMs

Uncertainty evaluation examines methods for quantifying and mitigating uncertainty in large language models to improve models’ reliability and decision-making robustness [30]. Probability-based metrics, often involving predictive confidence or output token entropy, have been extensively studied [7,18,31,32]. Additionally, consistency-based methods posit that LLMs produce logically inconsistent responses when uncertain or hallucinating [19,33,34,35]. Moreover, recent findings have introduced internal states to detect hallucinated generations from LLMs [17,36,37,38]. Furthermore, some works have explored prompting LLMs to express uncertainty or confidence in their responses [10,28,39,40]. We conducted a systematic evaluation of multiple methods for detecting unknown questions using the Honesty dataset. And our findings reveal that the synergistic combination of explicit prompting strategies and implicit uncertainty quantification approaches represents a highly promising direction for substantially improving model “integrity” and overall reliability.

3. Dataset Construction

LLMs are trained on massive datasets that encompass vast amounts of knowledge across various domains, and the specifics of their training data are often opaque or not fully disclosed. Hence, it is challenging to precisely define the knowledge that lies beyond the scope of LLMs. However, as illustrated in Figure 2, once these models undergo pretraining and fine-tuning, their parameters, which encode their knowledge, become fixed and immutable. As a result, events or developments occurring after the training of LLMs are inherently outside their knowledge scope. Therefore, the construction of unknown questions follows a deterministic rule based on temporal knowledge boundaries rather than subjective annotation.

We have gathered recent events that occurred after the deployment of all LLMs involved in experiments, specifically between October 2024 and December 2024. Then, we manually formulate unknown questions of Honesty dataset based on the gathered information. Specifically, two researchers on our team collected news articles from multiple platforms and generated a set of related questions from the content. For instance, using information about the 59th Golden Bell Awards, they created the question: “Who won Best Actor at the 59th Golden Bell Awards?” Subsequently, to ensure the quality and effectiveness of the unknown questions, a third researcher filtered questions based on the following guidelines:

• Unknown questions should be clear and provide a factual answer, avoiding ambiguity. For example, the question “What type of event was disrupted by protesters in New York on Thanksgiving Day?” does not specify which year’s Thanksgiving Day is being referred to. Consequently, the answer may be within LLMs’ parametric knowledge.
• The generated questions should refrain from using terms such as “recent,” “latest,” or other expressions that imply ambiguous temporal references, as LLMs don’t know the real-world date. So the questions framed in this manner should not be deemed beyond the model’s knowledge boundary.

To guarantee a comprehensive evaluation, we opted for “known questions” drawn from the SelfAware dataset [23]. Since the requisite information to answer these questions is available on Wikipedia, a foundational component of the pre-training corpus for contemporary LLMs, it is plausible to infer that the model possesses sufficient knowledge to answer them. As shown in

Table 1. Examples of known and unknown questions in the Honesty dataset.

Field	Known Question	Unknown Question
question_id	36	729
question	“Who was the first human to take a step on the Moon?”	“Who has won the Best Actor at 59th Golden Bell Awards?”
answer	[“armstrong”]	null
known	true	false
source	“SelfAware”	“Unknown”

, the data in the Honesty dataset contains five fields, with the “known” field indicating whether the item is known or unknown.

Due to the labor-intensive nature of manually crafting questions, we initially generated 1000 unknown questions. After filtering and deduplication, 502 high-quality unknown questions were retained. Combined with 681 known questions extracted from other sources, the Honesty dataset contains a total of 1183 entries. This scale is comparable to that of previous studies [23] and is fully sufficient for robust evaluation. Additionally, to ensure data diversity, the unknown questions we collected span multiple domains, including film, entertainment, military, technology, and more.

It is worth noting that DeepSeek-v3 was released in December 2024, yet its knowledge only extends up to July 2024 (as confirmed through direct queries to the model). In our subsequent experiments, we randomly sampled 50 responses from DeepSeek-v3 to unknown questions and found that all of them contained factual inaccuracies. This further highlights that the knowledge required to answer these unknown questions goes beyond the model’s knowledge boundaries. We conducted identical experiments on the GPT-4o model to mitigate the potential impact of any undisclosed internal updates implemented by the provider.

4. Experiment

4.1. Models

We conduct extensive experiments to evaluate the “integrity” of multiple LLMs, including GPT-4o [3], DeepSeek-V3 [15], and the Llama series [4,13,14], specifically involving Llama-7B, Llama-13B, Llama-30B, Llama-2-7B, Llama-2-7B-chat, and Llama-3.1-8B. And we perform a thorough analysis of the experimental results from various perspectives. For generations, all LLMs use the top-p/top-k sampling strategy; we set top-p to 0.99 and top-k to 10. We configured DeepSeek-V3 to 1.3, following the official recommendation, whereas the remaining models were set to 0.7. The number of generations is set to 10.

4.2. Evaluation Method

We use the automated methodology proposed in [23] to determine whether the responses generated by LLMs conveyed uncertainty. Specifically, we first devise a collection of reference sentences that explicitly express a lack of confidence in the answers to the current question, such as “I don’t know the answer” or “That’s beyond my knowledge scope,” etc (detailed further in Appendix A). Then we utilize a similarity function to compute the similarity, $sim$, between the generated answer, S, of a question, and the reference sentences set $U=\{u_1,u_2,\dots ,u_n\}$:

$$sim=\underset{i=1,2,3,\dots ,n}{max}\left(SimCSE(S,u_i)\right)$$

(1)

Whenever $sim$ surpasses a pre-determined threshold $\tau$, we regard this as the model successfully conveying uncertainty, meaning that it recognizes the current item as an unknown question. In the experiments, we employed SimCSE [41] to calculate the similarity. We choose SimCSE because it provides high-quality sentence embeddings and consistently achieves strong performance on semantic similarity tasks. Its contrastive learning framework enables it to capture fine-grained semantic relations, making it well-suited for distinguishing uncertainty expressions from regular responses. In addition, SimCSE is lightweight and easy to integrate, allowing reliable similarity estimation without task-specific training. In our implementation, we directly use the princeton-nlp/sup-simcse-bert-base-uncased model from Hugging Face.

Following the study [23], we first generate a pool of candidate uncertainty expressions and manually filter those that genuinely express uncertainty. These human-validated sentences serve as positive examples for calibrating the similarity-based detection. Subsequently, we employ SimCSE with different similarity thresholds and evaluate detection performance under each setting. And

Table 2. Evaluation results under different thresholds.

SimCSE	Precision (%)	Recall (%)	F1 (%)
0.60	85.58	100.00	92.23
0.65	88.89	98.88	93.62
0.70	91.49	96.63	93.99
0.75	95.51	95.51	95.51
0.80	95.95	79.78	87.12
0.85	96.77	67.42	79.47
0.90	94.44	38.20	54.40

exhibits that the highest F1 score was achieved with a threshold of 0.75, while maintaining a great balance between precision and recall. Therefore, during experiments, we set $\tau$ to 0.75, and we adopt the F1 score as a measure of LLMs’ “integrity.” Since our focus is on identifying unknown questions, we treat them as positive cases during evaluation, while known questions are treated as negative cases.

4.3. Results and Analysis

4.3.1. Direct Prompting

As shown in Figure 3, this input form feeds a raw question as the prompt to LLMs (detailed prompts can refer to Figure A1 in Appendix B). As demonstrated in Figure 4, regardless of the model size and capability, all LLMs exhibited extremely low F1 scores. Furthermore,

Table 3. More detailed comparisons of LLMs across precision, recall, and confusion matrix metrics under different prompting strategies.

Model	Strategy	TP	FP	TN	FN	Precision	Recall
Llama-2-7B-chat	Direct	22	3	678	480	0.88	0.04
	Instruction	313	258	423	189	0.55	0.62
	ICL	242	120	561	260	0.67	0.48
	COT	412	85	596	90	0.83	0.82
DeepSeek-V3	Direct	30	2	679	472	0.94	0.06
	Instruction	271	24	657	231	0.92	0.54
	ICL	260	14	667	242	0.95	0.52
	COT	348	10	671	154	0.97	0.69
GPT-4o	Direct	33	1	680	469	0.97	0.07
	Instruction	342	32	649	160	0.91	0.68
	ICL	321	20	661	181	0.94	0.64
	COT	370	18	663	132	0.95	0.74

reveals that the models demonstrate virtually no recall on unknown questions. This suggests that even advanced models such as GPT-4o and DeepSeek-V3 struggle to proactively express uncertainty when they lack knowledge. To further validate this observation, we randomly sampled model responses under this setting for manual inspection and found that they predominantly consisted of confident but factually incorrect answers, with little to no explicit acknowledgment of uncertainty. LLMs still encounter substantial reliability challenges in real-world applications, underscoring the need for alignment with “integrity.”

4.3.2. Instruction Prompting

As shown in Figure 3, we concatenate the instruction and the question as prompts, guiding the LLMs to explicitly indicate when they lack the necessary knowledge (detailed prompts are provided in Figure A2 in Appendix B). Figure 4 illustrates improvements to varying degrees in the F1 scores of all LLMs compared to using direct prompting. This indicates that LLMs have the potential for “integrity,” but when presented only with raw questions, they tend to provide overly confident answers rather than expressing uncertainty.

Moreover, all LLMs that underwent fine-tuning, including Llama-2-7B-chat, GPT-4o, and DeepSeek-V3, showed significant improvements in F1 scores compared to models that were only pre-trained, with GPT-4o achieving the highest score of 78%. It is worth noting that we found that most responses from pre-trained models could not express uncertainty through self-awareness. Instead, they merely reproduced the uncertainty words from the instructions in a mechanical manner (several cases are in Appendix C). These indicate that the capability to explicitly express uncertainty of LLMs is predominantly shaped during the fine-tuning stage.

4.3.3. “Yes” or “No” Choice Prompting

Figure 4 also reveals differences in F1 scores across different fine-tuned models. Llama-2-7B-chat exhibits substantially lower performance compared to GPT-4o and DeepSeek-V3. This motivated us to investigate additional factors that may influence the “integrity” of LLMs. To further explore this, we conducted a series of experiments using Llama-2-7B-chat.

First, we observed that Llama-2-7B-chat sometimes failed to follow instructions properly, resulting in nonsensical outputs (see the Appendix C for detailed cases). Therefore, we transformed the original queries into “Yes” or “No” choice questions to prompt the model. We designed three types of prompting formats, as shown in Figure 5. This prompting form ensures the Llama-2-7B-chat model generates responses aligned with the instructions, thereby avoiding discrepancies in experimental results due to non-compliance with the instructions. The results in Figure 6 and Figure 7 illustrate that using the third choice prompting brought Llama-2-7B-chat’s F1 score to the same level as DeepSeek-V3. This demonstrates that the model’s ability to follow instructions, which can also be considered a broader semantic understanding capability, is a critical factor affecting its “integrity.”

Additionally, as illustrated in Figure 5, the F1 score of the third prompting format reached 67%, which is significantly higher than the first format by several percentage points. This suggests that in order to build reliable models, we should not simply align them with a general “know/don’t know” distinction. Instead, breaking down the problem into finer-grained aspects might yield better results.

Another interesting finding is that when the model is asked, “Do you know the answer to the question or do you need more information?”, its F1 score drops to 10%, almost eliminating its “integrity.” This suggests that unequal options in the question may lead the large model to lean towards denying its own ignorance.

4.3.4. COT Prompting

So far, the highest F1 score of the Llama-2-7B-chat model is 67%, still exhibiting a significant gap compared to GPT-4o’s 78%. Based on our analysis, Llama-2-7B-chat exhibits a propensity for erroneous judgments when confronted with questions requiring multi-step reasoning or arithmetic computations. For example, see Figure 8, the model knows that “Meiko has never played for IG,” but when asked about Meiko’s teammates in IG, it still confidently made things up.

Thus, we aim to use the COT prompting format to enhance the reasoning capabilities of Llama-2-7B-chat. The experimental results in Figure 6 illustrate that Llama-2-7B-chat’s F1 score has significantly improved, reaching 82% and surpassing GPT-4o with instruction. Compared with other prompting forms, it exhibits a significant performance improvement. This demonstrates that reasoning ability plays a crucial role in influencing the “integrity” of LLMs (detailed cases are provided in the Appendix C). However, when using only in-context learning (ICL) prompting, the F1 score drops to just 56%. The diagram of the two prompting strategies is illustrated in Figure 9 (detailed prompts are provided in Figure A3 and Figure A4 in Appendix B). Compared to ICL, the COT examples incorporate not only a conclusive uncertainty statement but also a sequence of intermediate rationales that lead to this conclusion. These intermediate rationales, while enhancing the reasoning abilities of LLMs, also provide explicit thought patterns to teach models to identify scenarios where they lack knowledge.

As depicted in Figure 7, the performance of all three models under ICL prompting is comparable to that under Instruction prompting. In contrast, CoT prompting consistently yields noticeable improvements over other forms. Additionally, from Table 3, we can observe that CoT prompting outperforms both Instruction and ICL prompting not only in improving recall for unknown questions but also in substantially reducing false positives for known questions. This indicates that CoT prompting can effectively guide models to balance recognizing knowledge insufficiency with avoiding unnecessary misclassification of known information, thereby demonstrating that CoT prompting more effectively elicits the “integrity” of LLMs and represents a promising approach for enhancing their reliability. To confirm that the observed gains are not due to random variation, we performed paired bootstrap tests (1000 resamples) on all instances using Llama-2-7B-Chat. Compared to direct prompting, COT yields a highly significant improvement of $\Delta$F1 = 0.763 (95% CI: [0.722, 0.800], p < 0.001). Against instruction prompting, CoT again significantly outperforms the baseline ($\Delta$F1 = 0.265, 95% CI: [0.225, 0.304], p < 0.001). These results confirm that the superiority of CoT is statistically robust and highly significant.

Furthermore, we investigate whether prompting LLMs to explicitly express uncertainty affects their performance on known questions. As shown in

Table 4. Impact of different prompting formats on known question answering accuracy.

Strategy	Llama-2-7B-chat	DeepSeek-V3	GPT-4o
Direct	0.82	0.89	0.89
Instruction	0.55	0.84	0.83
ICL	0.66	0.89	0.89
COT	0.78	0.89	0.89

, Instruction prompting significantly degrades accuracy on known questions across all three models, with varying degrees of decline. In contrast, both ICL and CoT prompting leave the performance of DeepSeek-V3 and GPT-4o completely unaffected. For Llama-2-7B-Chat, CoT induces a noticeably smaller drop compared to the other two strategies. These results further underscore the superiority of the CoT prompting strategy. Additionally, Llama-2-7B-Chat exhibits considerably larger performance fluctuations across prompting conditions than the other two models, indicating that models with stronger semantic understanding capabilities also display greater robustness to variations in prompt format.

4.3.5. Probability/Consistency Based Methods

In this section, we evaluate the effectiveness of several representative probability-/consistency-based uncertainty evaluation methods for detecting unknown questions and assess their performance in this context. In our experiments, we utilized three representative methods, including INSIDE [17], LN-Entropy [18], and Lexical Similarity [19]. INSIDE uses an internal state-level consistency measurement across multiple generations, called “EigenScore”, to detect uncertainty in LLMs. LN-Entropy is designed to measure sequence-level uncertainty by utilizing multiple generations, while Lexical Similarity evaluates the average similarity across multiple answers as a consistency measure. Here, a higher EigenScore and LN-Entropy indicate greater uncertainty, whereas a lower Lexical Similarity value corresponds to a higher degree.

Figure 10 illustrates the specific uncertainty quantification distributions of the three methods on the Llama-2-7B-chat model, and the results indicate that the model exhibits higher uncertainty when answering unknown questions. We then used three methods to detect unknown questions, and the results in Figure 11 demonstrate that all three uncertainty quantification methods achieve reasonable effectiveness across different models. Notably, compared to prompting-based methods, they also demonstrate some level of performance on LLMs that have only undergone pretraining. Consistent with prompting-based approaches, higher scores are observed on fine-tuned Llama-2-7B-chat, with the LN-Entropy method achieving an F1 score of 69%. The score is comparable to the model’s performance with instruction prompting, but it is far below the F1 score achieved with COT prompting as shown in Figure 6.

Finally, we tried combining COT prompts with these probability/consistency-based metrics to jointly detect unknown questions. The approach is as follows: first, the COT prompt is used to determine whether the current question is unknown. If not, one of the three methods is used for assessment. Only when both methods agree that the question is not an unknown question is it classified as a known question. And with this method, the F1 score achieved its highest value to date presented in Figure 6. This suggests that combining explicit and implicit strategies could be a promising approach to enhancing the reliability of LLMs. Perhaps more sophisticated combination methods, such as investigating optimal weights for integrating the two approaches, could lead to improved outcomes. However, this is not the focus of our paper, so we did not explore it further.

5. Conclusions

In this work, we explore the capacity of LLMs to identify questions beyond their knowledge boundaries. To achieve this, we manually curated a high-quality dataset named Honesty comprising both known and unknown questions. Using this dataset, we systematically evaluate the effectiveness of prompting-based techniques as well as several probability-based and consistency-based methods for detecting unknown questions. Experimental results reveal that certain qualitative behaviors—such as the tendency to hallucinate when knowledge is insufficient and the benefits of explicit guidance—are observed across both earlier and more recent models. Notably, reasonable prompting strategies, such as Chain-of-Thought (COT), can effectively stimulate the “integrity” of LLMs. Moreover, integrating with probability- or consistency-based methods yields further performance gains. These findings highlight the promising potential for aligning LLMs with honest behavior through targeted interventions. In future research, we will extend our analysis to a broader range of more recent and advanced LLMs, as well as pursue more comprehensive evaluations, including fine-tuning-based alignment and the study of verbalized confidence (e.g., self-rating scales), to deepen our understanding of model integrity. We hope our research can inspire future studies focused on developing more reliable and trustworthy LLMs, ultimately advancing the field and improving the practical applications of these models across various real-world scenarios.

6. Limitations

In the automated evaluation of model responses, we rely on a predefined set of reference uncertainty expressions to determine whether a model explicitly acknowledges a lack of knowledge. While this design follows prior work and provides a controlled evaluation setting, it may not capture all possible linguistic variations of uncertainty. Our prompting strategies, such as instruction and CoT prompting, explicitly guide models to express uncertainty using the reference expressions. In addition, our evaluation does not rely on strict string matching but instead employs semantic similarity, allowing lexically different yet semantically equivalent uncertainty expressions to be correctly identified. Despite these mitigations, relying on a finite set of reference expressions remains an inherent limitation of the current evaluation paradigm. Exhaustively enumerating all possible ways to express uncertainty is practically infeasible. Therefore, future work could explore methods for automatically discovering reference expressions of uncertainty or developing more robust and flexible automated evaluation techniques.

Additionally, our dataset Honesty primarily focuses on question-answering tasks and does not encompass other more complex natural language processing tasks, such as summarization, mathematical and logical reasoning, and code generation. Therefore, in future research, we plan to incorporate more diverse data sources and apply data augmentation and optimization tailored to different task requirements.