Class Entity Identification Based on Large Language Models: A Choice Between Classification and Generation ^†

Abstract

Large language models (LLMs) have been widely applied to knowledge graph question answering (KGQA) systems. Recent Text-to-SPARQL studies have demonstrated that generation performance can achieve an F1 score exceeding 90%. Further error analysis has categorized common errors into entity translation errors, entity position errors, and resource description framework (RDF) triple-count errors, with the latter accounting for 24% of all errors. Notably, nearly 90% of RDF triple-count errors occur when the triples involve class entities. Previous research has shown that incorporating prompts can effectively enhance model performance. Based on the results, we predicted whether a question contains a class entity and the number of RDF triples in the corresponding query to reduce RDF triple-count errors in large language models by providing precise task-related information through prompt design. Since both strategies are classification-oriented, two implementation paradigms were established: traditional classification architectures and generative modeling. They were compared in terms of performance. For classification-based architectures, we employed Bidirectional Encoder Representations from Transformers (BERT) and the Robustly Optimized BERT Approach (RoBERTa) to obtain question embeddings for classification. For the generative approach, we adopted the Instruction-Tuned Text-to-Text Transfer Transformer (Flan-T5). Experimental results show that the generative model slightly outperforms conventional classification architectures, indicating that generative approaches can achieve higher prediction accuracy and provide more reliable information without the need for additional complex encoder designs, thereby improving the overall quality of Text-to-SPARQL generation.

1. Introduction

Previous studies [1,2] have demonstrated that task-oriented input–output format design, when combined with an appropriate model, can effectively enhance the quality of Text-to-SPARQL generation. Although some approaches have achieved F1 scores exceeding 90% [1,2,3], these studies also categorize Text-to-SPARQL errors into three major types: entity translation errors, entity position errors, and RDF triple-count errors.

Figure 1 illustrates an example question from the Large-Scale Complex Question Answering Dataset (LC-QuAD) and its corresponding SPARQL Protocol and RDF Query Language (SPARQL) query [4]. This query retrieves the answer to the question by executing multiple RDF triples, each consisting of a subject, predicate, and object. RDF triples involving class entities are characterized using the modifier <rdf:type>, with <dbo:River> serving as an example of a class entity in the figure. The following examples demonstrate the three common error types in Text-to-SPARQL generation.

• Entity translation error: SELECT DISTINCT ?uri WHERE {?uri <dbo:River> <dbr:Lake_Ontario>. ?uri <rdf:type> <dbo:River>.}
• Entity position error: SELECT DISTINCT ?uri WHERE {<dbr:Lake_Ontario> <dbo:Source> ?uri. ?uri <rdf:type> <dbo:River>.}
• RDF triple-count error: SELECT DISTINCT ?uri WHERE {?uri <dbo:Source> <dbr:Lake_Ontario>.}

Figure 1. Example from the LC-QuAD.

Figure 1. Example from the LC-QuAD.

In the first case, <dbo:source> is incorrectly translated as <dbo:River>, representing an entity translation error. In the second, the subject (<dbr:Lake_Ontario>) and object (?uri) are reversed, leading to an entity position error. In the third, a missing triple results in an RDF triple-count error.

According to the error analysis by Chen et al. [2], RDF triple-count errors account for approximately 24% of all errors. The investigation in this study shows that nearly 90% of these errors occur when the RDF triples involve class entities, that is, when one or more triples containing <rdf:type> are missing or redundant. Inspired by recent advances in prompt-based large language models [5,6,7,8,9], we explore whether such methods can reduce these errors. Prompts aim to provide appropriate and accurate task-related information (including instructions, examples, or knowledge) to lower the likelihood of generation errors. Therefore, we developed a system that automatically constructs prompts that provide this essential information.

Based on the characteristics of the conceptual design of RDF triples, we propose two prompt-generation strategies: predicting whether a question contains a class entity and the number of RDF triples required for the query. For the LC-QuAD [4] used in this study, each query contains at most one triple involving a class entity. Hence, the first method determines whether the question includes such a triple. The second method, motivated by the observation that triple-count errors usually involve missing or redundant triples, aims to predict the required number of triples. In future work, both types of information may be integrated to further minimize triple-count errors.

Since the proposed tasks exhibit a clear correspondence between input and output, similar to entity type classification [10], we formulated them as classification problems. Traditional classification tasks are commonly built on encoder-based architectures [10], where the encoder generates semantic representations for classification. As shown in Figure 2a, features extracted by large language models are projected into a classifier to produce corresponding labels.

Figure 2. Comparison of the encoder and encoder–decoder architectures.

Generative modeling has become increasingly popular in recent years. Unlike encoders, encoder–decoder models such as Flan-T5 [11] directly generate sequence-based outputs, as illustrated in Figure 2b. Although generative models have been successfully applied to tasks like sentiment analysis [11], few studies have directly compared them with traditional encoder-based classifiers. Therefore, in this work, we not only aim to automatically construct prompts but also to compare the effectiveness of traditional classification models and generative approaches for our designed tasks.

2. Method

In this study, for the two proposed methods, we implemented and compared encoder-based and generative strategies to evaluate how different model architectures affect sentence-level classification performance. For the encoder-based architecture, as illustrated in Figure 2a, the question is first processed by the encoder to obtain contextualized features. These token-level representations are then aggregated through a pooling operation to form a sentence-level semantic representation [12]. The resulting vector is passed to a classifier for prediction. Depending on the specific task, the output label sets are defined as follows:

• Class entity prediction: $y\in \{0, 1\}$, where 1 indicates that the question contains a class entity.
• RDF triple-count prediction: $y\in \{1, 2, 3, 4\}$, representing the required number of RDF triples in the corresponding SPARQL query.

In this research, BERT [13], RoBERTa [14], and Flan-T5 encoder were used as the backbone models. For the classifier, we adopted a standard linear classification layer, which maps the sentence representation to the output label space.

For the generative architecture, Flan-T5 was employed as the core model, as illustrated in Figure 2b. The input sentence is first encoded into semantic features by the encoder, and the decoder subsequently generates task-specific textual outputs. In the class entity prediction task, the model generates either “1” or “0,” indicating the presence or absence of a class entity. In the RDF triple-count task, the model produces one of “1,” “2,” “3,” or “4,” representing the predicted number of RDF triples required for the SPARQL query.

Inspired by the examples provided in Ref. [10], we designed two types of prompt templates, each differing in the way they constrain or describe the expected answer. As shown in Table 1, these templates demonstrate how varying the phrasing of the prompt can lead to distinct input sentences. The examples are constructed using the sample question and corresponding SPARQL query shown in Figure 1, illustrating the flexibility of prompt-based generation under different formulations.

Table 1. Input–output design of the generative tasks.

Task	Template	Example	Output
Class type	Class entities required: 0 or 1 Q: {q} A:	Class entities required: 0 or 1 Q: What is the river whose source is Lake Ontario? A:	1
	Answer the following 0/1 question. Q: {q} A:	Answer the following 0/1 question. Q: What is the river whose source is Lake Ontario? A:	1
Triple number	Number of RDF triples required: 1, 2, 3, or 4 Q: {q} A:	Number of RDF triples required: 1, 2, 3, or 4 Q: What is the river whose source is Lake Ontario? A:	2
	Answer the following question about RDF triple count. Q: {q} A:	Answer the following question about RDF triple count. Q: What is the river whose source is Lake Ontario? A:	2

3. Results and Discussion

In this study, we adopted the LC-QuAD developed by Trivedi et al. [4] as the experimental dataset. The original dataset contains 4001 training questions and 1000 test questions. Since LC-QuAD includes several complex question types, such as comparative and superlative questions, that are beyond the scope of this study, we excluded those instances and removed duplicate questions. After preprocessing, 3977 questions remained in the training set and 991 questions in the test set.

In the training set, 1549 questions contain class entities, while 2428 questions do not. Regarding the distribution of RDF triple counts, there are 1084 questions with one triple, 1665 questions with two triples, 1225 questions with three triples, and only 3 questions with four triples. In the test set, 349 questions contain class entities, whereas 642 questions do not. As for the RDF triple distribution, 276 questions contain one triple, 439 questions contain two triples, and 276 questions contain three triples.

3.1. Comparison of Encoder Architectures

For the encoder-based experiments, we evaluated BERT, RoBERTa, and the Flan-T5 encoder, which serves as the core baseline of this study. The results are shown in Table 2. As expected, RoBERTa achieved the highest accuracy across both tasks. RoBERTa is an improved version of BERT, trained on a much larger corpus, approximately 160 billion tokens, compared with BERT’s 16 billion. Although Flan-T5 was trained on even larger datasets, its encoder performed slightly worse than RoBERTa. This may be attributed to the fact that the Flan-T5 encoder was optimized primarily for use in conjunction with its decoder, rather than as a standalone encoder for classification tasks.

Table 2. Accuracy under the encoder architecture.

Model	Accuracy
	Class Type	Triple Number
BERT	0.8587	0.8244
RoBERTa	0.8648	0.8365
FlanT5 Encoder	0.8628	0.8325

3.2. Comparison of Generative Methods

The experimental results for the generative approach are presented in Table 3. For the class entity prediction task, both of the designed prompt templates produced nearly identical results, with accuracy values that were either very close or completely identical. This finding indicates that, within our experimental design, prompt phrasing did not significantly affect model performance.

Table 3. Accuracy under the generator architecture.

Model	Accuracy
	Class Type	Triple Number
FlanT5 (v1)	0.8698	0.8416
FlanT5 (v2)	0.8698	0.8426

3.3. Comparison Between Encoder and Generative Approaches

As shown in Table 4, we compared the Flan-T5 generative model with its encoder-only counterpart and also included the best-performing encoder-based model (RoBERTa) for reference. The results demonstrate that the generative approach consistently outperformed both the standalone Flan-T5 encoder and the best encoder-based model overall. This outcome suggests that even with simple prompt formulations, the generative architecture can achieve higher predictive accuracy than traditional encoder-based models with linear classifiers. Consequently, the findings confirm that prompt-based generative modeling offers a more effective and flexible solution for sentence-level classification tasks related to Text-to-SPARQL generation.

Table 4. Comparison of the accuracy of the encoder and the generator.

Model	Accuracy
	Class Type	Triple Number
RoBERTa	0.8648	0.8365
FlanT5 encoder	0.8628	0.8325
Flan T5	0.8698	0.8426

4. Conclusions

We compared encoder-based and generative strategies on two proposed classification tasks. The experimental results show that, under a baseline setting without any specially designed encoder structure, the generative model achieved slightly higher accuracy than the encoder-based baseline. This finding suggests that, even without additional architectural design efforts, the generative approach can serve as an effective and ready-to-use alternative that provides better overall performance. At the same time, this result highlights a promising direction for future research: further exploring the upper performance limits of encoder-based methods and comparing them systematically with generative approaches.

The generative strategy can be considered a practical and robust short-term choice for achieving strong performance quickly, while long-term optimization may benefit from developing task-aligned and structurally enhanced encoder-based models, which could potentially yield even better results through a more specialized design.