KLSBench: Evaluating LLM Capabilities on Korean Literary Sinitic Texts in Historical Context

Featured Application

KLSBench provides essential evaluation protocols for deploying LLM-powered applications on digitized Korean Literary Sinitic archives. This benchmark enables systematic assessment of automated annotation systems, intelligent search interfaces, and educational platforms for digital humanities projects, supporting historians, philologists, and students in accessing and analyzing historical text collections.

Abstract

Large language models (LLMs) show limited capability in processing low-resource historical languages due to insufficient training data and domain-specific linguistic structures. Korean Literary Sinitic (KLS), the principal written medium of the Joseon dynasty, remains particularly under-resourced despite its lexical overlap with modern Korean and shared script with classical Chinese. To enable systematic evaluation in this domain, we introduce KLSBench, a comprehensive benchmark for assessing LLM performance on KLS. KLSBench contains 7871 instances sourced from Joseon dynasty civil service examination archives and parallel corpora of the Four Books, and encompasses five task categories: classification, retrieval, punctuation restoration, natural language inference, and translation. Our evaluation suggests KLSBench could work as an effective diagnostic tool that distinguishes lexical recall from deeper linguistic comprehension in low-resource historical languages. Beyond establishing evaluation baselines, KLSBench provides practical frameworks for deploying LLM-based tools in digital humanities contexts, including automated annotation systems and intelligent search interfaces for classical text repositories.

1. Introduction

Large language models (LLMs) [1,2,3,4] have demonstrated impressive performance on a wide range of natural language processing tasks [5,6,7] and benchmarks, particularly for modern high-resource languages [8,9]. However, the ability of LLMs to generalize from high-resource modern benchmarks to low-resource historical languages remains poorly understood, particularly in scenarios where training data is limited and linguistic conventions differ substantially from modern usage. Addressing this limitation is important for evaluating the practical applicability of LLMs to historical and low-resource language processing tasks.

A key challenge in studying this problem is the lack of appropriate evaluation benchmarks. Existing multilingual and historical benchmarks often focus on modernized or translated texts, making it difficult to isolate whether LLMs can leverage cross-lingual knowledge when direct supervision is limited [10,11,12]. As a result, we currently lack systematic evidence on how LLMs perform when faced with historical languages that are low-resource yet structurally connected to high-resource languages.

Korean Literary Sinitic (KLS; see Appendix C for a definition) serves as an appropriate testbed for addressing this gap. Historically used in Korea as the primary written language for scholarship and administration, KLS is currently low-resource in terms of digital corpora, while maintaining close linguistic connections to modern Chinese through shared logographic characters and to modern Korean through extensive Sino-Korean vocabulary (see Appendix C for a definition) [13,14]. This combination of data scarcity and cross-lingual relatedness enables systematic evaluation of whether LLMs can leverage cross-lingual information to mitigate limited in-language resources, thereby making KLS well suited for examining cross-lingual generalization in low-resource historical language settings.

Beyond data scarcity, KLS also poses linguistic challenges that further stress current models. Texts are written in condensed classical syntax with implicit grammatical relations [15] and are typically preserved without punctuation (白文; see Appendix C for a definition), requiring models to infer sentence boundaries and structure. Moreover, KLS texts contain dense cultural references and rhetorical conventions rooted in classical East Asian traditions [16]. These characteristics distinguish KLS from modern language benchmarks and highlight the need for evaluation settings that reflect the realities of historical text processing.

To evaluate LLM performance under these conditions, we introduce KLSBench, a benchmark designed to assess understanding of Korean Literary Sinitic. Following the design philosophy of $C_3$Bench [17] for Chinese classical texts, KLSBench is constructed from Joseon dynasty civil service examination materials and canonical Confucian texts from the Four Books (四書). The benchmark comprises 7871 instances across five tasks—classification, retrieval, punctuation, natural language inference, and translation—covering both linguistic form and semantic understanding. We evaluate seven LLMs, including proprietary models (GPT-4 Turbo, GPT-3.5 Turbo, Claude 3.5 Sonnet, Claude 3 Opus) and open-source models (LLaMA 3.1, Qwen 2.5, EXAONE 3.0). Our results reveal clear performance stratification across task types, offering new insights into the strengths and limitations of current LLMs in leveraging cross-lingual knowledge for low-resource historical languages.

The main contributions of this work are as follows:

1.

We introduce KLSBench, the first comprehensive benchmark for evaluating LLM performance on Korean Literary Sinitic, which contains 7871 instances across five task categories, constructed from Joseon civil service examination materials and parallel corpora of the Four Books.

2.

We conduct a systematic evaluation of seven LLMs covering general-purpose, Chinese-specialized, and Korean-specialized pretraining profiles. The evaluation results provide a quantitative performance landscape for KLS understanding, enabling reproducible comparison across model families and establishing a public leaderboard.

3.

We publicly release KLSBench along with evaluation code and detailed documentation to facilitate reproducible research and continued advancement in Korean Literary Sinitic understanding. The benchmark, code, and zero-shot outcomes are available at https://songhune.github.io/korean_R-CoA/ (accessed on 10 January 2026).

Section 2 reviews related work on LLM evaluation benchmarks and classical text processing in historical languages. Section 3 presents the construction of KLSBench, including data sources, task design for all five tasks, and dataset statistics. Section 4 describes the experimental setup, evaluated models, and evaluation metrics. Section 5 presents comprehensive results and analyses for each task, including overall performance, model comparisons, and error analysis. Section 6 discusses key findings, implications for classical text understanding, and limitations. Finally, Section 7 concludes the paper and outlines directions for future research.

2. Related Works

2.1. LLM Evaluation Benchmarks

Large language model development has driven creation of evaluation benchmarks to assess capabilities across tasks and domains. MMLU (Massive Multi-task Language Understanding) [8] evaluates models across 57 subjects including humanities, STEM, and social sciences. HELM (Holistic Evaluation of Language Models) [9] provides a framework covering accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency across 42 scenarios. BigBench [18] extends evaluation with over 200 tasks contributed by the research community.

Domain-specific benchmarks evaluate specialized knowledge and reasoning. Legal-Bench [19] assesses legal reasoning across tasks including case analysis and contract understanding. Medical benchmarks such as MedQA [20] and PubMedQA [21] evaluate medical question answering using USMLE-style questions and biomedical literature. ScienceQA [22] provides multimodal science question answering across physics, chemistry, and biology.

Multilingual evaluation has received attention [11,23]. XTREME [24] and XGLUE [25] benchmark cross-lingual understanding and generation across 40+ languages, focusing on modern high-resource languages. XNLI [26] extends natural language inference evaluation to 14 languages including low-resource languages. For Korean, KorNLI and KorSTS [27] evaluate natural language inference and semantic textual similarity in contemporary Korean, while KLUE [28] provides a benchmark suite for modern Korean language understanding.

2.2. Low-Resource and Classical Language Processing

Evaluating LLMs on classical and historical languages presents challenges beyond modern language processing. Cao et al. [17] introduced $C_3$Bench, a classical Chinese understanding benchmark comprising approximately 50,000 instances across five tasks spanning ten domains of classical literature. Their evaluation of 15 LLMs revealed performance gaps between classical and modern Chinese understanding, showing that proficiency in contemporary language does not automatically transfer to classical texts. This work established baselines for classical Chinese evaluation, yet evaluating classical texts in other East Asian contexts remains unexplored.

Literary Sinitic text processing involves difficulties distinct from modern language tasks [29,30,31]. The absence of explicit word boundaries necessitates segmentation algorithms [15], while named entity recognition must account for historical figures, places, and institutions no longer in contemporary use [16,32]. Classical texts employ grammatical structures and literary conventions that differ from modern forms [33], requiring linguistic knowledge for interpretation. These challenges are compounded by limited training data [34], absence of modern native speakers, and sparse linguistic resources [35,36].

Korean classical texts present challenges beyond $C_3$Bench for Chinese classical texts [17]. KLS is a low-resource language with connections to high-resource languages. Traditional Literary Sinitic using orthodox characters (正體字) has been largely excluded from contemporary Chinese NLP research following mainland China’s standardization of simplified characters (字) in the 1950s–1960s. However, KLS has connections to two high-resource languages: modern Chinese and modern Korean. Modern Chinese is historically and lexically connected to KLS through their shared use of the logographic character system and substantial overlap in classical lexical forms, while modern Korean has inherited a substantial body of vocabulary from Sino-Korean etymological roots. This creates a scenario where transfer learning from related high-resource languages could compensate for training data scarcity—yet whether LLMs exploit these cross-lingual bridges remains unverified.

Korean’s dual-script nature (Hangul and Literary Sinitic, detailed explanation described in Appendix C), character encoding complexities (multiple Unicode normalization forms, polyphone characters like 更and 度, initial-sound law variations), and region-specific cultural interpretations require specialized preprocessing. No evaluation framework exists for Korean classical literature understanding. Establishing such a benchmark addresses this gap and helps understand how LLMs leverage cross-lingual knowledge for low-resource historical languages with connections to contemporary high-resource languages.

3. KLSBench: Benchmark Construction

Figure 1 provides an overview of the KLSBench architecture from data sources through evaluation. The benchmark integrates two data sources (Joseon dynasty Gwageo records and Four Books) through a data processing pipeline. We evaluate five tasks using seven LLMs and produce performance analysis. Task examples are provided in the Appendix A.1.

3.1. Data Sources

KLSBench uses two primary data sources drawn from the Joseon dynasty’s literary and examination traditions.

3.1.1. Joseon Dynasty Civil Service Examination Records

The Gwageo (科擧) civil service examination system selected government officials during the Joseon dynasty, operating from 1392 until its abolition in the Gabo Reforms of 1894. The examination consisted of multiple stages and required mastery of KLS literature, philosophy, and literary genres. The dataset draws from historical examination records including original questions and model answers preserved in literary collections and examination anthologies.

We extract data from three classes in the knowledge graph: Question, Answer, and Sigwon (physical answer sheets). After preprocessing including duplicate removal, length filtering, format standardization, and quality verification, we retained 2849 instances from an original corpus of 3348 question–answer pairs. Each instance includes the original KLS text with Korean and English translations, spanning 21 literary genres. Figure 2 shows the temporal distribution of examination questions across the Joseon dynasty, with concentration in later periods due to better preservation.

3.1.2. Four Books (四書)

The Four Books (Saseo: 論語Analects, 孟子Mencius, 大學Great Learning, 中庸Doctrine of the Mean) are the canonical texts of Neo-Confucian education and served as the foundation of the Gwageo examination curriculum. These texts, part of the Four Books and Five Classics (四書五經) corpus, contain Confucian philosophical concepts and ethical principles that shaped intellectual discourse throughout the Joseon dynasty.

We use trilingual editions of the Four Books published by the Korean Institute for Classical Humanities [37,38,39], based on the Jipju (集註) tradition by Zhu Xi (朱熹, 1130–1200), which served as the official text for Joseon civil service examinations. We extract data from Work (KLS original texts) and Collection (Korean and English translations with commentaries) classes in the knowledge graph. After preprocessing including duplicate removal, segmentation standardization, and quality verification, we obtained 2119 instances from an original corpus of 2624 passage–translation–commentary triplets.

Table 1. Data source composition and KLSBench task utilization.

Source	Data Class	Content	Tasks
Gwageo Records (2849)	Question	Test prompts with	Classification
		21 literary genres	(808)
	Answer	Model responses	Punctuation
		from anthologies	(1000)
	Sigwon	Historical answer	Translation
		sheets with grades	(1320)
Four Books (2119)	Work	Literary Sinitic	Retrieval
		original texts	(1209)
	Collection	Korean/English	NLI (1854),
		translations	Punctuation
		with commentaries	(1000),
			Translation
			(680)
Total Instances			4968
Coverage			All 5 tasks

summarizes the composition and utilization of both data sources across KLSBench tasks.

3.2. Task Design

KLSBench comprises five tasks to evaluate different dimensions of classical literature comprehension. Each task addresses capabilities required for understanding historical texts, ranging from surface-level text processing to semantic reasoning. Examples of each task are provided in Appendix A. (

Table A3. Classification task examples.

Example 1—Philosophical Discourse (論)

Input: 子曰學而時習之不亦說乎有朋自遠方來

不亦樂乎人不知而不慍不亦君子乎

Expected Output: 論(Ron, Discourse)

Rationale: This passage exhibits characteristic

features of philosophical discourse: didactic tone,

rhetorical questions, and moral reasoning about

learning and self-cultivation.

Example 2—Historical Narrative (記)

Input: 太祖在位三年秋七月朝鮮國王遣使來朝貢

方物其使曰臣國自古事大以誠今聞天子即位故遣

臣來獻

Expected Output: 記(Gi, Record)

Rationale: Contains temporal markers,

chronological structure, and narrative of historical

events involving diplomatic relations.

Example 3—Expository Essay (說)

Input: 夫水之性清而土汩之故不清人之性善而

物誘之故不善是以君子必慎其獨也

Expected Output: 說(Seol, Exposition)

Rationale: Analytical argumentation using

analogical reasoning (water metaphor) to explain

abstract philosophical principles.

–

Table A6. Natural language inference task examples.

Example 1—Entailment

Premise (KLS): 子曰學而時習之不亦說乎

Hypothesis (Korean): 공자께서 배움과

복습의 기쁨에 대해 말씀하셨다

Label: Entailment

Justification: The hypothesis accurately

captures the semantic content of Confucius

discussing the joy of learning and practice.

Example 2—Neutral

Premise (KLS): 子曰克己復禮為仁

Hypothesis (Korean): 공자께서 예의범절을

지켜야 한다고 말씀하셨다

Label: Neutral

Justification: While related, the hypothesis

focuses narrowly on ritual propriety without

capturing the core concept of self-discipline or

humaneness.

Example 3—Contradiction

Premise (KLS): 子曰人不知而不慍不亦君子乎

Hypothesis (Korean): 공자께서 남이 알아주지

않으면 화를 내야 한다고 하셨다

Label: Contradiction

Justification: The hypothesis directly

contradicts the premise’s assertion that a noble

person remains unperturbed by lack of recognition.

).

3.2.1. Classification Task

Objective: the classification task evaluates LLMs’ ability to identify literary genres (文體; see Appendix C for a definition) in KLS texts. During the Joseon dynasty, examinations required mastery of literary genres, each governed by distinct conventions regarding structure, argumentation, and stylistic features. Accurate classification requires understanding genre-specific characteristics rather than recognizing superficial textual patterns.

Data Construction: we construct a dataset of 808 instances spanning 21 literary genres extracted from Gwageo examination records. The major genres include bu (賦, rhyme-prose), si (詩, regulated poetry), eui (疑, essays on doubtful points), ui (義, argumentative essays), chaek (策, policy proposals), pyo (表, memorials), ron (論, discourses), myeong (銘, inscriptions), and jeon (箋, annotations). We employ stratified sampling with 95 instances per major genre, supplemented by samples from minor genres. Each instance consists of KLS text extracted from either the abstract or full content of examination questions and answers.

Evaluation: we report accuracy as the primary metric, supplemented by macro-averaged precision, recall, and F1 scores to account for class imbalances across the 21 genres.

3.2.2. Retrieval Task

Objective: the retrieval task assesses LLMs’ ability to identify the source text and chapter from which a KLS passage originates. This task simulates source attribution and tests models’ familiarity with canonical texts that formed the foundation of Joseon dynasty education.

Data Construction: we construct 1209 instances from the Four Books, as these texts are well-defined canonical works with established chapter structures. The distribution reflects the relative sizes of the source texts: Lunyu (論語, Analects, 500 instances), Mengzi (孟子, Mencius, 500 instances), Zhongyong (中庸, Doctrine of the Mean, 137 instances), and Daxue (大學, Great Learning, 72 instances). Each instance requires the model to output both the book title and the chapter from which the passage is drawn.

Evaluation: we measure accuracy based on exact match of both book title and chapter name. Partial credit is not awarded, as precise source attribution requires identifying both components correctly.

3.2.3. Punctuation Task

Objective: Literary Sinitic texts were traditionally written without punctuation marks (白文), requiring readers to segment sentences and clauses based on syntactic and semantic understanding. The punctuation restoration task evaluates LLMs’ ability to insert punctuation marks into unpunctuated KLS texts, testing their comprehension of classical grammar, sentence boundaries, and logical structure.

Data Construction: we create 2000 instances by removing punctuation from KLS texts. The dataset maintains balanced representation from both data sources: 1000 instances (50%) from Four Books and 1000 instances (50%) from Gwageo examination records. Preprocessing includes filtering for minimum text length (10 characters or more). We preserve only content-bearing punctuation marks (periods, commas, question marks) used in modern scholarly editions while removing formatting punctuation. This task requires models to understand KLS syntax and discourse structure to identify clause and sentence boundaries.

Evaluation: we adopt character-level F1 score as the primary metric. We treat punctuation restoration as a sequence labeling task. Additionally, we report ROUGE-1, ROUGE-2, and ROUGE-L scores to capture n-gram overlap and longest common subsequence similarity between predicted and reference punctuated texts.

3.2.4. Natural Language Inference Task

Objective: Natural language inference (NLI) evaluates LLMs’ ability to determine logical relationships between pairs of statements. Given a premise and hypothesis, models must classify the relationship as entailment (the hypothesis logically follows from the premise), contradiction (the hypothesis contradicts the premise), or neutral (no logical relationship exists).

Data Construction: we generate 1854 NLI instances using construction strategies to ensure coverage of inference types. The label distribution reflects natural occurrence patterns: entailment (1313 instances, 70.8%), neutral (400 instances, 21.6%), and contradiction (141 instances, 7.6%). We employ six generation categories:

1.

Translation equivalence: pairing original KLS with Korean translations

2.

Cross-lingual entailment: connecting Korean and English translation pairs

3.

Cross-text relation: comparing passages from different texts or chapters

4.

Negation-based: introducing logical negation to create contradictions

5.

Metaphorical reasoning: testing understanding of figurative language

6.

Analogical reasoning: evaluating comprehension of analogies and parallels

This approach ensures the task assesses reasoning capabilities beyond semantic similarity matching.

Evaluation: we report overall accuracy as the primary metric, complemented by per-class precision, recall, and F1 scores for each of the three labels (entailment, neutral, contradiction) to identify specific reasoning weaknesses.

3.2.5. Translation Task

Objective: the translation task evaluates LLMs’ multilingual comprehension and generation across three languages: KLS, Korean, and English. Translation requires lexical knowledge and understanding of cultural context, philosophical concepts, and stylistic conventions across linguistic boundaries.

Data Construction: we construct 2000 translation instances covering two language pair directions. Literary Sinitic to Korean translation (1320 instances, 66%) constitutes the majority, reflecting the importance of rendering classical texts into vernacular Korean. Korean-to-English translation (680 instances, 34%) tests the ability to convey Korean interpretations of classical concepts into English. The asymmetric distribution mirrors translation practice priorities in classical Korean studies, where vernacularization of KLS remains the primary activity.

Evaluation: we employ BLEU score [40] as the primary evaluation metric. This metric measures n-gram precision between model outputs and reference translations. We additionally report ROUGE-1, ROUGE-2, and ROUGE-L scores [41] to capture recall-oriented similarity and lexical overlap. Translation quality assessment requires human expert judgment, which we use for qualitative error analysis.

Table 2. Examples of translation challenges in KLSBench.

Example 1: KLS to Korean

Source (KLS): 子曰學而時習之不亦說乎有朋自遠方來不亦樂乎

Target (Korean): 공자께서 말씀하시기를, 배우고 때때로 익히면 또한 기쁘지 아니한가. 벗이 먼 곳에서 찾아오니 또한 즐겁지 아니한가.

Challenge: Preserving classical philosophical tone while conveying rhetorical questions in modern Korean.

Example 2: Korean to English

Source (Korean): 孟子께서 말씀하시기를, 사람은 모두 남을 차마 해치지 못하는 마음이 있다.

Target (English): Mencius said, "All people have a heart that cannot bear to see the suffering of others."

Challenge: Maintaining philosophical register and capturing nuanced compassion concepts in English academic style.

Example 3: Culturally Embedded Terms

Source (KLS): 克己復禮為仁一日克己復禮天下歸仁焉

Target (Korean): 자기를 이기고 예로 돌아가는 것이 인을 행하는 것이니, 하루라도 자기를 이기고 예로 돌아가면 천하가 인으로 돌아올 것이다.

presents examples illustrating translation challenges in KLSBench. The first example demonstrates KLS-to-Korean translation, where condensed classical syntax (子曰學而時習之不亦說乎) requires expansion in the target language to convey implicit subjects and philosophical nuances. The second example shows Korean-to-English translation, requiring preservation of formal register while capturing culturally specific concepts of compassion (차마 해치지 못하는 마음). The third example highlights the challenge of translating core Neo-Confucian terminology (克己復禮, 仁) that resists direct lexical equivalence and requires cultural contextualization rather than word-by-word correspondence.

3.3. Data Statistics and Quality Control

3.3.1. Overall Statistics

Table 3. KLSBench overall statistics.

Task	Instances	Percentage	Metrics
Classification	808	10.3%	Acc, P, R, F1
Retrieval	1209	15.4%	Accuracy
Punctuation	2000	25.4%	F1, ROUGE
NLI	1854	23.6%	Acc, P, R, F1
Translation	2000	25.4%	BLEU, ROUGE
Total	7871	100%	-

presents statistics for KLSBench, showing the distribution of instances across all five tasks and their evaluation metrics.

3.3.2. Label Distribution

For tasks requiring categorical prediction, we provide detailed label distribution statistics.

Table 4. Label distribution for classification and NLI tasks.

Task	Label	Count	Percentage
Classification 1	賦(Bu)	95	11.8%
	詩(Si)	95	11.8%
	疑(Eui)	95	11.8%
	義(Ui)	95	11.8%
	策(Chaek)	95	11.7%
	表(Pyo)	95	11.7%
	論(Ron)	51	6.3%
	銘(Myeong)	53	6.6%
	Others (13 categories)	134	16.5%
NLI	Entailment	1313	70.8%
	Neutral	400	21.6%
	Contradiction	141	7.6%

¹ Percentages may not sum to exactly 100% due to rounding.

shows the distribution for Classification and NLI tasks, which exhibit different balance characteristics.

The Classification task contains 21 literary genres with imbalanced distribution. Six major genres (詩, 表, 策, 義, 疑, 賦) contain exactly 95 instances each through stratified sampling, while the remaining 15 minor genres range from 2 to 53 instances, reflecting the natural frequency distribution of classical literary genres in examination records. The NLI task displays natural label imbalance. Entailment constitutes the majority (70.8%), which reflects the prevalence of translation equivalence pairs in our construction methodology.

3.3.3. Text Length Statistics

Text length variation impacts both task difficulty and computational requirements.

Table 5. Text length statistics across tasks (in characters).

Task	Mean	Median	Min	Max
Classification (input)	324	198	12	3847
Retrieval (input)	156	98	8	1245
Punctuation (input)	187	142	10	1892
NLI (premise)	142	108	6	982
NLI (hypothesis)	138	105	6	976
Translation (source)	165	127	8	1456

presents length statistics across all tasks, measured in characters for Korean text and tokens for KLS.

The Classification task exhibits the highest mean length (324 characters) and maximum length (3847 characters). This reflects the inclusion of complete examination essay responses. Retrieval and NLI tasks feature shorter texts (mean 142–156 characters). These correspond to canonical philosophical passages from the Four Books. This length diversity ensures the benchmark tests comprehension capabilities across varying context windows.

Figure 3 illustrates text length distributions across major literary genres in the classification task. This demonstrates that different literary genres exhibit characteristic length patterns. For instance, bu (賦, rhyme-prose) and chaek (策, policy proposals) tend toward longer compositions, while si (詩, regulated poetry) maintains relatively uniform shorter lengths due to strict formal constraints.

3.3.4. Quality Control Measures

We implement rigorous quality control procedures to ensure benchmark reliability and validity:

Duplicate Detection and Removal. We identify and eliminate duplicate instances using exact text matching and near-duplicate detection based on character-level edit distance. From initial corpora of 3348 Gwageo records and 2624 Four Books passages, we removed 499 and 505 duplicates, respectively. We retained 2849 and 2119 unique instances.

Length Filtering. We apply minimum length thresholds to exclude trivially short instances that provide insufficient context for meaningful evaluation. Specifically, we filter instances shorter than 10 characters for all tasks. This ensures that remaining examples contain substantive content requiring actual comprehension rather than simple pattern matching.

Format Standardization. To ensure stable processing of KLS text, we apply Unicode Normalization Form C (NFC) to all instances. This step resolves inconsistencies arising from heterogeneous character sources, particularly when a visually identical graphic like ‘女(female)’ can be encoded either as U+5973 or (in Korean-compatibility mode) as U+F981. In historical corpora annotated under Korean orthographic variants (due to the 頭音法則—“initial-sound rule”) the divergence of initial-sound representations (‘녀’ vs. ‘여’) often co-occurs with differing codepoints rather than purely pronunciation difference. Without normalization, a tokenizer may treat U+5973 and U+F981 (or variant sequences) as distinct symbols, causing artificial sparsity, segmentation mismatches, or embedding misalignment. NFC consolidation maps such variants to a canonical form (typically U+5973), thus unifying initial-sound rule-related and encoding-derived variants, and ensuring consistent downstream processing of characters that are linguistically identical.

Label Validation. For Classification and Retrieval tasks derived from structured databases, we validate label accuracy by cross-referencing with original source metadata. For NLI instances generated through automated strategies, we conduct manual verification of a random 10% sample (185 instances). We achieve 94.6% agreement with intended label assignments. Disagreement cases primarily involved subtle neutral-entailment boundary judgments in cross-text relations.

Translation Quality Verification. Reference translations from the Korean Institute for Classical Humanities [42] are officially published scholarly materials, providing a reliable baseline for the Translation task.

These quality control measures ensure that KLSBench provides reliable, high-quality evaluation data suitable for systematic benchmarking of LLM capabilities on Korean classical literature understanding.

4. Experimental Setup

4.1. Evaluated Models

We evaluate seven large language models, including proprietary API-based systems and open-source models. These models are grouped by access mode (proprietary vs. open-source) and linguistic pretraining backgrounds. This selection covers a range of model families and training profiles under a unified evaluation setting. Across both proprietary and open-source models, we standardize the query format and prompting procedure following the empirical inference protocol of Yamaguchi et al. [43] to ensure consistent and reproducible evaluation inputs.

Table 6. Specifications of evaluated models.

Model	Type	Parameters	Release
Proprietary API Models
GPT-4 Turbo	API	Unknown	2024
GPT-3.5 Turbo	API	Unknown	2023
Claude 3.5 Sonnet	API	Unknown	2024
Claude 3 Opus	API	Unknown	2024
Open-Source Models
Llama 3.1 Instruct	Open	8B (Billion)	2024
Qwen 2.5 Instruct	Open	7B	2024
EXAONE 3.0 Instruct	Open	7.8B	2024

summarizes the specifications of all evaluated models.

Proprietary Models. We evaluate four proprietary models accessed through their official APIs. GPT-4 Turbo and GPT-3.5 Turbo (OpenAI) [1] represent the GPT-4 and GPT-3.5 families and are evaluated via the OpenAI API. Claude 3.5 Sonnet and Claude 3 Opus (Anthropic) represent the Claude 3 family and are accessed through the Anthropic API. All proprietary models are evaluated using default inference configurations provided by their respective APIs. Due to their closed-source nature, architectural details and parameter counts are not publicly disclosed.

Open-Source Models. We evaluate three open-source instruction-tuned models [44,45] in the 7–8B parameter range. This narrow parameter range is chosen to control for model size and to support a more balanced comparison among open-source models under local inference settings. Llama 3.1 8B Instruct (Meta) [2] is a general-purpose multilingual model. Qwen 2.5 7B Instruct (Alibaba Cloud) [3] is trained with a strong emphasis on Chinese-language data. EXAONE 3.0 7.8B Instruct (LG AI Research) [46] is developed using Korean-language resources. All open-source models are evaluated using their official Hugging Face implementations with identical instruction-following templates and decoding settings.

4.2. Evaluation Metrics

We employ task-specific evaluation metrics aligned with established practices in NLP benchmarks. All metrics are computed using standard implementations from scikit-learn (classification metrics) and NLTK/rouge-score libraries (generation metrics).

4.2.1. Classification and Retrieval Tasks

For Classification and Retrieval tasks, we report the following:

• Accuracy: Proportion of correctly predicted labels, serving as the primary metric.
• Macro-averaged F1: Harmonic mean of precision and recall averaged across all classes, addressing potential class imbalance.
• Per-class Precision and Recall: Class-specific metrics to identify strengths and weaknesses across different literary genres (classification) or source texts (retrieval).

4.2.2. Punctuation Restoration Task

For the Punctuation task, we treat punctuation insertion as a sequence labeling problem and evaluate the following:

• Character-level F1 Score: Primary metric measuring precision and recall of punctuation mark placement at the character level.
• ROUGE-L F1: Longest common subsequence-based similarity between predicted and reference punctuated texts, capturing overall structural preservation.
• ROUGE-1 and ROUGE-2 F1: Unigram and bigram overlap metrics, respectively, providing complementary assessment of local accuracy.

4.2.3. Natural Language Inference Task

For NLI, we report the following:

• Overall Accuracy: Primary metric measuring correct classification across all three labels.
• Per-class F1 Scores: Separate F1 scores for entailment, neutral, and contradiction to diagnose model behavior across different logical relationships.
• Confusion Matrix Analysis: Qualitative assessment of systematic misclassification patterns, particularly distinguishing between neutral-entailment and neutral-contradiction confusions.

4.2.4. Translation Task

For Translation, we employ multiple automatic metrics to capture different aspects of translation quality:

• BLEU Score: Primary metric measuring n-gram precision with brevity penalty, standard for machine translation evaluation.
• ROUGE-1, ROUGE-2, ROUGE-L F1: Recall-oriented metrics providing complementary assessment of lexical overlap and structural similarity.

We acknowledge that automatic metrics provide approximations of translation quality and conduct qualitative error analysis on sample translations to complement quantitative results.

4.3. Implementation Details

4.3.1. Hardware Configuration

All experiments are conducted on a single NVIDIA H100 GPU (80 GB VRAM) with 12-core CPU and 240 GB system memory (238 GB application memory, 2 GB shared memory). This configuration supports efficient inference for both API-based models (via network requests) and local deployment of open-source models. API models are accessed through official endpoints with standard rate limiting and retry mechanisms. Open-source models are loaded using PyTorch with bfloat16 precision for memory efficiency while maintaining numerical stability.

4.3.2. Prompt Design and Inference Strategy

We adopt a zero-shot evaluation paradigm to assess models’ inherent capabilities without task-specific examples. All prompts follow a consistent structure:

1.

System Prompt: Task-specific instructions clearly defining the expected input–output format and behavioral constraints.

2.

User Prompt: The actual test instance formatted according to task requirements.

For classification, we provide the KLS text and request identification of the literary style from a predefined list. For retrieval, we present a passage and ask for the source book and chapter. For punctuation, we provide unpunctuated KLS text (白文) and request insertion of appropriate punctuation marks to segment sentences and clauses. For NLI, we present premise–hypothesis pairs and request classification into entailment, neutral, or contradiction. For translation, we specify the source language, target language, and text to be translated.

In designing the prompt descriptions, we use the descriptive conventions widely used in the most established LLM benchmarks, GLUE [47]. Accordingly, each task includes a clear statement of its objective, a precise explanation of the input structure, explicit definitions of all output labels, and a concise summary of the evaluation protocol. Although such benchmarks do not impose a rigid template, they consistently provide these components, and we followed the same principles to ensure clarity, comparability, and reproducibility.

4.3.3. Inference Parameters

We standardize inference hyperparameters across all models to ensure fair comparison:

• Temperature: 0.0 (deterministic decoding for reproducibility).
• Top-p (nucleus sampling): 1.0 (disabled due to temperature = 0.0).
• Max output tokens: Task-dependent (classification/retrieval/NLI: 100; punctuation: 2000; translation: 1500).
• Random seed: 42 (for open-source models).

For API models, we use the latest stable API versions as of June–September 2025, including OpenAI API v1 and Anthropic API. For open-source models, we use Transformers library version 4.40.0 with PyTorch 2.2.0.

4.3.4. Reproducibility

To ensure reproducibility, we provide the following:

• Complete evaluation code and prompts available at the project repository;
• Fixed random seeds for all stochastic processes;
• API version specifications and model checkpoint identifiers;
• Detailed hyperparameter configurations;
• Raw model outputs for all evaluated instances.

All evaluation runs were conducted from June to September 2025, and API model behaviors are documented with their respective version timestamps to account for potential model updates. KLSBench will undergo regular updates in the future, including continuous evaluation of new LLM models, data quality improvements through expert validation, and expansion with additional tasks.

5. Results and Analysis

5.1. Overall Performance

Table 7. Overall performance across all tasks and models.

Model	Type	Classif.	Retrieval	Punct.	NLI	Trans.
		Acc	Acc	F1	Acc	BLEU
GPT-4 Turbo	API	0.327	0.758	0.768	0.827	0.307
GPT-3.5 Turbo	API	0.366	0.600	0.940	0.741	0.253
Claude Sonnet 4.5	API	0.113	0.917	0.406	0.005	0.329
Claude 3 Opus	API	0.440	0.808	0.473	0.362	0.250
Llama 3.1 8B	Open	0.156	0.233	0.839	0.741	0.189
Qwen 2.5 7B	Open	0.515	0.592	0.511	0.757	0.033
EXAONE 3.0 7.8B	Open	0.312	0.633	0.356	0.719	0.144
Mean		0.318	0.649	0.613	0.593	0.215
Std Dev		0.098	0.202	0.227	0.293	0.097

presents comprehensive performance across all seven evaluated models and five tasks. The results reveal dramatic performance variations both across tasks and across models. This demonstrates that classical literature understanding presents fundamentally different challenges from modern text processing.

5.2. Task-Specific Analysis

5.2.1. Classification Results

Literary style classification remains challenging, with mean accuracy of 31.8%. Performance shows substantial variation across models: Qwen 2.5 7B achieves the highest accuracy at 51.5%, dramatically outperforming Claude 3 Opus (44.0%), GPT-3.5 Turbo (36.6%), GPT-4 Turbo (32.7%), EXAONE 3.0 (31.2%), Llama 3.1 8B (15.6%), and Claude Sonnet 4.5 (11.3%). The performance range (11.3% to 51.5%) reveals substantial differences in classical genre recognition capabilities. This hierarchy demonstrates that Chinese-specialized pretraining provides decisive advantages: Qwen 2.5 7B substantially exceeds both general-purpose (ChatGPT, Claude) and Korean-specialized (EXAONE) models, revealing that exposure to classical Chinese texts during pretraining is critical for rhetorical genre recognition. The substantial performance gap between Qwen 2.5 7B’s 51.5% and the next-best Claude 3 Opus at 44.0% underscores the importance of classical corpus exposure.

The confusion matrix analysis (Figure 4) reveals systematic error patterns that illuminate the linguistic challenges of classical genre recognition. Models exhibit strong bias toward predicting 詩(poetry), the most structurally distinctive genre, resulting in high false-positive rates, with certain genres frequently misclassified as poetry. This pattern suggests models rely heavily on surface-level stylistic features (metrical patterns, parallelism) that characterize poetry but fail to recognize the deeper rhetorical structures distinguishing other genres. Conversely, 疑(essays on doubtful points) demonstrates relatively strong diagonal strength (56.4% recall), indicating that its distinctive question–answer dialectical structure provides more robust classification signals. The particularly weak diagonals for 賦(13.8%) and 義(13.3%) confirm that these genres’ subtle argumentative conventions—requiring understanding of classical rhetorical theory rather than surface textual patterns—remain largely inaccessible to current LLMs.

5.2.2. Retrieval Results

Retrieval demonstrates moderate performance (mean 64.9%), with considerable variation across models. Claude Sonnet 4.5 achieves the highest accuracy (91.7%), followed by Claude 3 Opus (80.8%), GPT-4 Turbo (75.8%), EXAONE 3.0 (63.3%), GPT-3.5 Turbo (60.0%), and Qwen 2.5 (59.2%). Notably, Llama 3.1 8B shows the lowest performance (23.3%), suggesting limited exposure to East Asian classical texts. The task relies on text matching and memorization rather than deep comprehension. The Four Books represent canonical texts extensively cited across East Asian intellectual history, likely appearing in various forms within LLM training data. Models can succeed through pattern matching without necessarily understanding philosophical content.

5.2.3. Punctuation Results

Punctuation restoration exhibits the most dramatic performance variation across models, with F1 scores [48] ranging from 0.356 to 0.940. Most remarkably, GPT-3.5 Turbo achieves an F1 score of 0.940, substantially outperforming all other models including the ostensibly more capable GPT-4 Turbo (0.768). Llama 3.1 (0.839) also demonstrates strong performance, while Qwen 2.5 (0.511), Claude 3 Opus (0.473), Claude Sonnet 4.5 (0.406), and EXAONE 3.0 (0.356) lag significantly behind.

5.2.4. Natural Language Inference Results

NLI performance reveals surprisingly strong reasoning capabilities, with mean accuracy of 59.3%. GPT-4 Turbo achieves the highest performance at 82.7%, followed by Qwen 2.5 (75.7%), GPT-3.5 Turbo (74.1%), Llama 3.1 (74.1%), EXAONE 3.0 (71.9%), and Claude 3 Opus (36.2%). Critically, Claude Sonnet 4.5 achieves near-zero accuracy (0.5%), suggesting severe limitations in its instruction-following or reasoning capabilities for this specific task format.

The NLI task requires determining logical relationships between premise–hypothesis pairs drawn from classical philosophical texts. This demands comprehension of logical structure and philosophical concepts. The strong performance of GPT-4 Turbo and several other models suggests that modern LLMs possess substantial formal reasoning capabilities over KLS semantic content. However, Claude Sonnet 4.5’s catastrophic failure (0.5%) despite strong retrieval performance (91.7%) reveals that NLI success depends critically on task-specific instruction tuning rather than general language understanding.

5.2.5. Translation Results

Translation demonstrates uniformly low but variable performance, with BLEU scores ranging from 0.033 to 0.329. Claude Sonnet 4.5 leads at 0.329, followed by GPT-4 Turbo (0.307), GPT-3.5 Turbo (0.253), Claude 3 Opus (0.250), Llama 3.1 (0.189), EXAONE 3.0 (0.144), and Qwen 2.5 (0.033).

These low BLEU scores reflect fundamental challenges in classical-to-modern language translation. Literary Sinitic employs highly condensed syntax and culturally embedded idioms requiring substantial interpretive work. Reference translations often include explanatory glosses absent from model outputs, yielding low n-gram overlap despite potential semantic adequacy. The relatively consistent performance suggests improvements require advances in cross-lingual understanding of classical concepts rather than model-specific optimizations.

5.3. Model Comparison and Key Insights

Models demonstrate task-specific strengths but struggle to achieve balanced performance across all dimensions, indicating that classical literature understanding comprises multiple orthogonal skill sets. Individual models excel at specific tasks: Claude Sonnet 4.5 achieves exceptional retrieval (91.7%), GPT-3.5 Turbo dominates punctuation (94.0%), GPT-4 Turbo leads NLI (82.7%), and Qwen 2.5 7B achieves the highest classification accuracy (51.5%). However, no single model maintains strong performance across all five tasks. Critically, classification remains challenging across all models (mean 31.8%), though Qwen 2.5 7B (51.5%) substantially outperforms all competitors, demonstrating the decisive advantage of Chinese-specialized pretraining for classical genre recognition. The substantial performance gap between Qwen (51.5%) and Claude 3 Opus (44.0%) reveals that classical corpus exposure during pretraining provides critical benefits, while EXAONE’s moderate performance (31.2%) demonstrates meaningful but limited cross-lingual transfer from Korean. Open-source models show competitive NLI performance (Qwen 75.7%, Llama 74.1%, EXAONE 71.9%), challenging assumptions that reasoning tasks require proprietary models, while classification results reveal that pretraining corpus composition substantially impacts understanding of Neo-Confucian rhetorical taxonomy.

5.4. Temperature Ablation Study

To assess robustness of model performance to generation parameters, we conduct a temperature ablation study at settings 0.0 (deterministic), 0.3 (moderate), and 0.7 (high) for GPT-3.5 Turbo, GPT-4 Turbo, Claude 3 Opus, and Claude Sonnet 4.5. All other hyperparameters remain fixed as specified in Section 4.

Table 8. Temperature sensitivity analysis: model performance across temperature settings.

Model	Task	T = 0.0	T = 0.3	T = 0.7
GPT-3.5 Turbo	Classification	0.366	0.386	0.290
	Retrieval	0.600	0.642	0.633
	Punctuation	0.940	0.941	0.936
	NLI	0.741	0.728	0.735
	Translation	0.253	0.243	0.230
GPT-4 Turbo	Classification	0.327	0.351	0.305
	Retrieval	0.758	0.758	0.775
	Punctuation	0.768	0.770	0.775
	NLI	0.827	0.822	0.834
	Translation	0.307	0.307	0.298
Claude 3 Opus	Classification	0.440	0.403	0.000
	Retrieval	0.808	0.808	1.000
	Punctuation	0.473	0.465	0.000
	NLI	0.362	0.379	0.000
	Translation	0.250	0.124	0.000
Claude Sonnet 4.5	Classification	0.113	0.125	0.100
	Retrieval	0.917	0.917	0.900
	Punctuation	0.406	0.415	0.404
	NLI	0.005	0.010	0.000
	Translation	0.329	0.334	0.329

shows that most models exhibit stable performance across temperature settings. Commercial models (GPT-3.5, GPT-4, Claude Sonnet 4.5) demonstrate robust performance with mean absolute deviation below 0.03 for most task–model combinations, validating our use of temperature = 0.0 for main evaluations. GPT-3.5’s punctuation (0.940) and GPT-4’s retrieval (0.758) remain nearly constant across all temperatures.

Claude 3 Opus shows higher temperature sensitivity, with performance degrading at T = 0.7 due to increased stochasticity disrupting token-level precision. Retrieval demonstrates the highest stability across all models, confirming this task’s reliance on pattern recognition rather than generation quality. These findings confirm that observed performance differences reflect genuine capability gaps rather than parameter-dependent artifacts, and that the task difficulty hierarchy (retrieval > NLI > punctuation > translation > classification) remains consistent across temperature settings.

6. Discussion

6.1. Key Findings from KLSBench Evaluation

Our evaluation reveals three critical insights for low-resource historical language processing. First, pretraining corpus composition critically determines genre recognition capabilities: the Chinese-specialized model substantially outperforms Korean-specialized alternatives for classical genre classification, demonstrating that exposure to KLS texts during pretraining provides decisive advantages despite KLS’s Korean lexical elements. Second, logical reasoning capabilities transfer robustly across domains: multiple models achieve strong NLI performance despite limited classical Chinese exposure, suggesting that formal inference abilities generalize beyond training distribution. However, catastrophic failures in specific models reveal brittleness in instruction-following. Third, task difficulty stratification reflects distinct cognitive demands: lexical matching (retrieval) proves substantially easier than genre recognition (classification), which requires understanding of Neo-Confucian rhetorical conventions and historical context.

6.2. Implications for Historical Language Processing

These findings carry practical implications for historical text processing. Current LLMs demonstrate utility for specific tasks like retrieval and punctuation assistance, with moderate capabilities for genre classification and logical inference. However, deploying LLMs for cultural–historical understanding requires careful model selection and expert verification. Future development should prioritize instruction-tuning integrating rhetorical theory and philosophical concepts beyond classical corpus exposure, exploring approaches including continued pretraining [5,49], retrieval-augmented generation [50,51], and multi-task learning [44,45].

6.3. Generality and Extensibility

While KLSBench is constructed for Korean Literary Sinitic, the benchmark design itself is not limited to this language. Korean Literary Sinitic constitutes a particularly informative case because it exhibits several properties commonly observed in historical and low-resource languages, including limited annotated data, condensed classical syntax, and sustained connections to high-resource modern languages through shared writing systems and lexical overlap [17,31,33]. These characteristics make KLS suitable for examining cross-lingual generalization, while remaining representative rather than exceptional among historical language settings.

More generally, the design of KLSBench follows principles that are not tied to any single language. First, the benchmark draws on historically authoritative and educational texts to ensure linguistic and semantic validity, a practice widely adopted in prior work on classical and historical language benchmarks [17,18]. Second, it includes a range of task types that assess both surface-level form understanding (e.g., punctuation and classification) and deeper semantic reasoning (e.g., inference and translation), in line with established multi-task benchmark design practices [25,47]. Third, the benchmark minimizes reliance on modernized preprocessing, allowing models to be evaluated on texts that remain close to their original historical form, consistent with prior studies on classical text processing [16,31]. Together, these design choices support application of the benchmark framework to other historical or low-resource languages with comparable textual traditions.

Accordingly, the KLSBench framework can be adapted to other settings, including classical variants of Chinese used in different regions, historical Japanese texts and other classical languages supported by long-standing scholarly corpora [17,33]. Although language-specific expertise is required for data curation and annotation, the overall benchmark structure, task formulation, and evaluation protocol remain transferable. From this perspective, KLSBench is intended not only as a benchmark for Korean Literary Sinitic, but also as a reference framework for the development of evaluation benchmarks for historical and low-resource languages more broadly.

7. Conclusions and Future Work

This study presents KLSBench, the first benchmark for KLS text understanding, comprising 7871 instances across five tasks. Our evaluation of seven LLMs reveals dramatic performance variations. KLSBench establishes evaluation protocols for deploying LLM-powered applications in digital humanities contexts. Beyond benchmark metrics, this work provides practical guidance for developing automated annotation systems, intelligent search interfaces, and educational platforms that can assist researchers in accessing and analyzing digitized historical text collections. KLSBench thus contributes both domain-specific evaluation infrastructure and actionable frameworks for building robust NLP applications on classical language resources.

7.1. Limitations

Benchmark Scope and Coverage. KLSBench comprises 7871 instances, smaller than $C_3$Bench (50,000 instances) for Chinese classical texts. This reflects availability of Korean historical text data, which is less digitized and annotated than Chinese literature. Our data sources are limited to two categories: Joseon dynasty civil service examination records and the Four Books. This focus excludes portions of KLS—including historical chronicles, private correspondence, Buddhist and Taoist texts, medical treatises, and administrative documents. Our benchmark primarily represents scholarly and canonical texts. Future expansion should incorporate broader coverage across genres and historical periods.

Language Pairs and Multilingual Evaluation. Our Translation task covers two language directions: KLS–Korean and Korean–English. This reflects historical practices in Korean classical studies but excludes language pairs: direct KLS–English translation, KLS–Chinese comparison (to assess differences from Chinese Literary Sinitic), and KLS–Japanese cross-lingual understanding (to compare with kanbun tradition). Our translation evaluation relies on automatic metrics (BLEU, ROUGE), which capture limited aspects of translation quality—lexical overlap and structural similarity rather than cultural appropriateness, stylistic fidelity, or philosophical nuance preservation. Human evaluation would provide more assessment of translation quality.

Evaluation Protocols and Settings. Our evaluation is limited to zero-shot settings, providing only task instructions to LLMs without in-context examples. This measures out-of-the-box model capabilities but does not assess whether few-shot learning could improve performance. Research shows few-shot prompting can bring gains on reasoning tasks, suggesting our NLI and classification results may be pessimistic. We use a single prompt template for all models, which may not be optimized for specific models or tasks. Prompt optimization would likely improve performance numbers though not change model rankings. Our automatic metrics (accuracy, F1, BLEU, ROUGE) provide scalable evaluation but do not fully capture human judgment. For generative tasks, especially translation and punctuation, human evaluation would provide more assessment of output quality.

Model Selection and Scope. We evaluate seven LLMs, but this selection is constrained by availability and computational resources. We do not evaluate larger open-source models (e.g., Llama 3.1 70B, Qwen 2.5 72B), limiting assessment of how model size scales to historical text understanding. We do not evaluate domain-specific models or models fine-tuned on KLS, as such models are not publicly available. Our evaluation is conducted from June–September 2025, and API model behaviors may change due to ongoing updates. Future work should evaluate a broader range of models, scale to larger model sizes, and explore fine-tuning and domain adaptation strategies.

Data Quality and Annotation Process. Our benchmark data derives from scholarly resources (Korean Institute for Classical Humanities, Academy of Korean Studies), which ensures high quality but limits our control over annotation processes. Classification labels derive from original metadata, which may reflect historical categorization rather than modern literary theory. NLI instances are generated through automated strategies and receive sample validation (94.6% agreement), but 5.4% disagreement indicates label noise. Translations are provided by the Korean Institute for Classical Humanities and officially published, but translation choices may vary across translators and periods, introducing subjectivity into translation evaluation.

Future work should improve annotation quality through expert validation, conduct inter-annotator reliability studies, and explore the incorporation of structured linguistic representations as complementary annotations for evaluating and supporting model behavior.

7.2. Future Research Directions

Our findings point to a methodological extension that directly addresses the structural challenges revealed by KLSBench. Korean Literary Sinitic is a highly formalized historical language characterized by stable syntactic and rhetorical regularities, including clause-based organization, parallel constructions, and genre-specific compositional schemes. Because such structure is not explicitly represented in the current benchmark, models are required to implicitly infer latent organization in tasks such as punctuation restoration, genre classification, and translation, which contributes to output instability and inter-genre confusion.

A natural extension of this work is to incorporate explicit structural representations as a complementary layer to probabilistic LLM outputs. This includes formalizing linguistic structure through clause segmentation, rhetorical templates, or relation-based representations derived through information extraction techniques (e.g., named entity and relation extraction). Such information can be organized into symbolic resources, including graph-based representations such as RDF-style subject–predicate–object triples, enabling explicit modeling of entities, relations, and hierarchical structure in KLS texts.

Under these conditions, hybrid inference paradigms that combine neural generation with post hoc structural validation offer a promising direction. In this paradigm, an LLM first generates multiple candidate outputs, which are subsequently filtered using KLS-specific syntactic or rhetorical constraints to ensure structural coherence and consistency. This approach has the potential to directly address the limitations observed in our zero-shot evaluation and provides a principled way to integrate generative flexibility with formal constraints.

Beyond the present evaluation, KLSBench opens several avenues for future research. Expanding benchmark coverage to additional historical genres would enable broader assessment across diverse textual traditions. Comparative studies across Literary Sinitic traditions may further clarify shared and localized properties of classical text processing. At the same time, developing human–AI collaborative frameworks remains essential for responsible deployment in digital humanities contexts. By providing systematic evaluation protocols and baselines, KLSBench establishes a foundation for such interdisciplinary exploration.