Enhancing Text-to-SPARQL Generation via In-Context Learning with Example Selection Strategies ^†

Abstract

Large language models demonstrate strong in-context learning (ICL) capabilities, allowing them to perform diverse tasks without fine-tuning. In knowledge graph question answering (KGQA), natural language questions are translated into SPARQL queries. Existing ICL approaches mainly rely on semantic similarity, often neglecting structural features. To address this limitation, we developed a structure-aware example selection strategy that integrates both semantic and structural patterns by abstracting Resource Description Framework (RDF) triples. We compare four strategies: (1) fully random, (2) semantic similarity, (3) same-type random, and (4) same-type semantic similarity. Experiments on LC-QuAD 1.0 using FLAN-T5 show that in non-fine-tuned settings, structure-aware semantic selection achieves the best results, highlighting the importance of structural congruence, while after fine-tuning, differences between strategies converge but diversity and semantic relevance remain beneficial. These findings demonstrate the critical role of example quality in ICL and provide empirical insights for KGQA design.

1. Introduction

In natural language processing (NLP), large language models (LLMs) have shown strong in-context learning (ICL) ability, enabling them to solve tasks such as question answering, translation, and text generation without fine-tuning [1,2,3,4]. Generative Pre-trained Transformer 3 (GPT-3), for example, demonstrates that ICL can approach or even surpass fine-tuned models in multi-task settings [1].

Knowledge graph question answering (KGQA) has emerged as a key research area. The core challenge lies in converting natural language questions into structured queries (e.g., SPARQL). Unlike conventional NLP tasks that generate free-form text or categorical labels, Text-to-SPARQL requires mapping natural language semantics to precise logical structures, strictly following syntax rules and composing RDF triples. As Zahera et al. highlighted, this process demands both accurate entity–relation alignment and syntactically valid queries [5]. Without structured prompts and ICL, LLMs often produce invalid SPARQL. These difficulties underline the complexity of Text-to-SPARQL beyond standard NLP tasks.

In recent studies, ICL is applied to Text-to-SPARQL [5,6,7,8]. However, most research relies on semantic similarity to select examples, overlooking structural features. For instance, Taffa and Usbeck [9] used k-Nearest Neighbors (kNN), which often faces scalability challenges. Zahera et al. reduced cost via clustering, but their method requires an appropriate number of clusters and often struggles with cross-dataset generalization [5]. Overall, these methods heavily rely on semantic similarity, leaving structural characteristics underexplored.

To address the limitations of reliance on semantic similarity, we introduce a structure-aware example selection strategy. Specifically, we transform SPARQL queries into RDF triples and abstract them into structural patterns (details and examples are provided in Section 3.1). We design fully random (FR), semantic similarity (SS), same-type random (STR), and same-type semantic similarity (STSS). Experiments on the LC-QuAD 1.0 dataset with the Flan-T5 model evaluate performance under zero-shot, one-shot, and few-shot settings and further compare ICL with fine-tuned models.

We conduct a systematic comparison of example selection strategies, proposing a structure-aware approach combined with semantic similarity; a systematic comparison of different example selection strategies is conducted under ICL. In addition, we analyzed the effects of ICL on both non-fine-tuned and fine-tuned models, providing design insights and empirical evidence for KGQA systems.

2. Related Works

ICL has been widely applied in NLP tasks without fine-tuning and is typically categorized into zero-shot, one-shot, and few-shot settings [1,10,11]. Previous research [2,12,13] shows that increasing the number of examples can improve performance. However, these gains yield diminishing returns and substantially higher computational costs. Consequently, the choice of example selection strategy becomes critical for ensuring both effectiveness and efficiency in ICL.

In Text-to-SPARQL, three representative strategies have been explored. Taffa and Usbeck applied kNN with cosine similarity to retrieve the nearest training questions [9]. Similarly, Zahera et al. employed clustering with Sentence Transformers, reducing computational cost but requiring pre-defined cluster numbers [5]. Kosten et al. compared random versus semantic similarity selection, showing that random examples sometimes outperform semantic ones due to greater diversity and reduced overfitting [8]. The results highlight the potential and the limitations of purely semantic-driven strategies, motivating our integration of structural features.

3. Methodology

Previous example selection methods mainly rely on semantic similarity, considering only the surface form of questions. To address this, we incorporate the structural characteristics of RDF triples to improve generation quality. This approach, different from clustering [5], does not require predefining the number of groups and thus provides clearer criteria and greater flexibility for optimization. In addition, random examples can sometimes mitigate overfitting from excessive semantic similarity [8], so we design two strategies within each structural type: random and semantic similarity.

3.1. Structural Classification of RDF Triples

To incorporate structural features into example selection, we adopted the structural abstraction framework of Chen et al. [14,15], simplifying SPARQL queries into RDF triples and further abstracting them into subgraph structural types.

For example, we consider the question “Which sectors’ people are part of local political parties that fall under the International Muslim Brotherhood?” (Figure 1). Its corresponding SPARQL query is expressed as three RDF triples: (?x, dbp:international, dbr:Muslim_Brotherhood), (?x, dbo:religion, ?uri), and (?x, rdf:type, dbo:PoliticalParty). In the abstraction process, dbr:Muslim_Brotherhood is treated as a named entity (E), dbp:international and dbo:religion are regarded as relation entities (R), and dbo:PoliticalParty as a class entity (C). The predicate rdf:type is likewise treated as a relation entity (R). Therefore, the triples are converted into the structural patterns (?x, R, E), (?x, R, ?uri), and (?x, R, C). After transformation, this RDF triple set contains one E, three R, and one C. Ignoring the directionality of R, the question is classified into the G-type structural category shown in Figure 2.

3.2. System Framework

Let the input question be denoted as $q$, whose vector representation is obtained from the final hidden state of the pre-trained Flan-T5 encoder. After flattening, a fixed-length embedding vector is generated, denoted as $e_q\in {\mathbb{R}}^d$, where $d$ represents the embedding dimension. Similarly, let the set of candidate questions extracted from the training data be $\mathcal{C}=\{c_1,c_2,\dots ,c_n\}$. Each candidate’s question $c_i$ undergoes the same encoding process to obtain its vector representation $e_{c_i}$.

The semantic similarity between the input question q and a candidate question c_i is then computed using cosine similarity as follows:

$$Similarity(c_i,q)=\mathrm{c}\mathrm{o}\mathrm{s}(e_q,e_{c_i})$$

(1)

Based on the similarity scores, the top-k most semantically similar candidate questions are selected, along with their corresponding RDF triples, to form the set $S_k$.

$$S_k=\left\{\left(c_i{,t}_i\right)|top\text{-}k\mathrm{by}Similarity(c_i,q)\right\}$$

(2)

where $t_i$ denotes the RDF triples associated with candidate question $c_i$.

During the prompt construction stage, the set $S_k$ is combined with the input question $q$ and fed into the Flan-T5 generation module to produce the target RDF triples as the final output.

3.3. Few-Shot Prompt Design

We evaluate the performance of large language models under three different ICL scenarios: zero-shot, one-shot, and few-shot and compare these settings with fine-tuned models. The few-shot prompt format is illustrated in Figure 3. In the experimental setup, the few-shot configuration consists of three reference examples together with one target question. Each reference example consists of a reference question, selected according to the example selection strategy, and its corresponding reference RDF triples. The reference triples are represented in a unified structured format with the aid of special tokens, which are designed to help the model correctly identify triple boundaries and variable dependencies. Specifically, [SEP] is used to separate the subject, predicate, and object; [ROOT] corresponds to the variable ?uri in the LC-QuAD 1.0 dataset; [TEMP] corresponds to the variable ?x, representing an intermediate node in the query; and [END] marks the termination of each triple.

3.4. Example Selection

To systematically investigate the roles of semantic similarity and structural features in example selection, and to validate the argument of Kosten et al. [8] that random examples may introduce useful diversity, we design and compare the following four strategies.

• FR: Examples are randomly sampled from the entire training dataset.
• SS: Both the input and training questions are encoded into vectors, and the most similar examples are selected based on cosine similarity.
• STR: The target question is first assigned to a structural type by the RDF triples classifier, and examples are randomly sampled within that category.
• STSS: The target question is assigned to a structural type, and the most semantically similar examples are selected from within the same category.

4. Experiments

We use Flan-T5 [16,17], which enhances generalization and reasoning in ICL scenarios. Experiments are conducted on the LC-QuAD 1.0 dataset to analyze how different ICL formats and example selection strategies affect generation accuracy.

4.1. Experimental Results Without Fine-Tuning

Table 1. BLEU and F1 scores in zero-shot, one-shot, and few-shot settings without fine-tuning.

Method	Setting	BLEU Score	F1 Score
FR	0-shot	0.0529	0.1566
	1-shot	0.1219	0.2372
	3-shot	0.1496	0.2423
SS	1-shot	0.2382	0.3563
	3-shot	0.3777	0.4913
STR	1-shot	0.1346	0.2505
	3-shot	0.2227	0.3296
STSS	1-shot	0.2329	0.3497
	3-shot	0.3980	0.5120

shows that model performance is limited under the zero-shot setting (BLEU = 0.0529, F1 = 0.1566). In contrast, few-shot settings significantly improve results. Among the four strategies, STSS shows the best 3-shot performance (BLEU score = 0.3980, F1 = 0.5120), indicating that structurally consistent and semantically relevant examples are most effective without fine-tuning. These findings suggest that the relevance and quality of examples have a greater impact on performance than merely increasing the number of examples.

4.2. Experimental Results with Fine-Tuning

Table 2. BLEU and F1 scores in zero-shot, one-shot, and few-shot settings with fine-tuning. (Bold values indicate the best performance in each column.)

Method	Setting	BLEU Score	F1 Score
FR	0-shot	0.8748	0.9158
	1-shot	0.8816	0.9188
	3-shot	0.8807	0.9198
SS	1-shot	0.8766	0.9177
	3-shot	0.8860	0.9224
STR	1-shot	0.8765	0.9175
	3-shot	0.8756	0.9156
STSS	1-shot	0.8763	0.9178
	3-shot	0.8783	0.9194

shows that overall performance improves substantially after fine-tuning. Even in the zero-shot setting, results are strong (BLEU score = 0.8748, F1 = 0.9158), indicating a solid baseline capability. In the 1-shot setting, FR performs marginally better (BLEU score = 0.8816, F1 = 0.9188), suggesting that example diversity can be more helpful than semantic or structural consistency in low-resource cases. With 3-shot, all strategies improve further, with SS achieving the best performance (BLEU score = 0.8860, F1 = 0.9224), followed by FR (BLEU score = 0.8807, F1 = 0.9198). This highlights the advantage of semantic relevance under richer conditions, while diversity from random selection still offers consistent benefits.

5. Conclusions

We analyzed the role of example selection strategies in ICL for RDF triple generation in KGQA. Beyond empirical validation, our results demonstrate that structural patterns can effectively guide large language models in structured query generation, and the relative importance of structural consistency, semantic similarity, and diversity evolves across training stages. These results enhance the theoretical understanding of ICL and offer practical guidance for building efficient, adaptive KGQA systems.