The Knowledge-Coherence Framework for Narrative Extraction: An Empirical Study on Scientific Literature

Abstract

Narrative extraction builds coherent ordered sequences of documents that trace how concepts develop over time, and is a growing area of information retrieval. In this work we focus on scientific literature, using a corpus of 3549 IEEE visualization research papers (1990–2022). A natural hypothesis is that augmenting embedding-based pathfinding with explicit domain knowledge should improve narrative quality. We present the Knowledge-Coherence Framework (KCF), which integrates structured metadata from OpenAlex into narrative extraction (building on the Narrative Trails algorithm), and conduct a systematic empirical investigation along three axes: (1) the effect of embedding model choice (MiniLM vs. SPECTER), (2) the effect of knowledge augmentation (with and without, plus sensitivity to the knowledge weight $\alpha$), and (3) the reliability of LLM-based evaluation (cross-agreement among 13 large language models). Throughout, mathematical coherence denotes the geometric mean of angular and topic similarity between consecutive documents along a path—an automatic, model-computed quantity inherited from Narrative Maps and Narrative Trails—while narrative quality refers to the LLM-judged construct. Using up to 600 evaluation pairs, we find that embedding model choice has a large effect on mathematical coherence (SPECTER: 0.94 vs. MiniLM: 0.81) and that, contrary to expectations, knowledge augmentation does not improve LLM-judged narrative quality—it slightly decreases it for both embeddings. Notably, the two notions dissociate: SPECTER produces the most mathematically coherent paths, yet MiniLM paths receive the highest LLM narrative-quality scores (5.87 vs. 5.36 out of 10). Alpha sensitivity analysis over five values ($\alpha \in \{0.0,0.3,0.5,0.7,1.0\}$, 500 pairs) confirms that LLM scores remain essentially flat while mathematical coherence steadily declines with increasing knowledge weight. Cross-model evaluation with 13 LLM judges shows high inter-model agreement (median Pearson $r=0.71$), supporting evaluation reliability. The main practical takeaways are that (i) embedding model choice, not knowledge augmentation, is the more consequential design decision, and (ii) mathematical coherence and LLM-judged narrative quality are distinct optimization targets that practitioners should not conflate.

1. Introduction

Constructing coherent narratives from large document collections is a challenge that cuts across news archives, historical records, scientific literature, and legal databases [1,2,3]. Traditional information retrieval (IR) systems are good at finding relevant documents, yet they fall short when tasked with assembling coherent narratives—ordered sequences of documents that tell meaningful stories about how concepts develop and interconnect over time [4].

Current narrative extraction methods, such as Narrative Trails [5], rely primarily on embedding similarity to keep consecutive documents coherent. The underlying assumption is that maximizing semantic similarity between adjacent documents yields the most coherent narrative. But when domain knowledge is available, conceptual connections may matter more than surface-level semantic similarity. To illustrate, consider a researcher exploring how “computer vision” and “natural language processing” connect in the literature: a purely embedding-based path might traverse documents with overlapping vocabulary yet miss conceptual bridges—papers on image captioning or multimodal learning—that actually link these two fields.

The reasoning above leads to what seems like a natural hypothesis: if we augment the pathfinding process with explicit domain knowledge, the resulting narratives ought to improve. We designed the Knowledge-Coherence Framework (KCF) to test exactly this claim. KCF layers structure metadata drawn from OpenAlex [6] on top of embedding-based narrative extraction for scientific literature, and we evaluate the framework—without presupposing that knowledge augmentation actually helps—along three axes:

1.

Embedding model choice: We compare general-purpose sentence embeddings (MiniLM [7,8]) against domain-specific scientific document embeddings (SPECTER [9]) across six method configurations.

2.

Knowledge augmentation: We evaluate whether incorporating OpenAlex concept metadata improves narrative quality, including a sensitivity analysis across five values of the knowledge weight parameter $\alpha$.

3.

LLM evaluation reliability: We assess the consistency of LLM-as-a-judge evaluation [10] using 13 different large language models, establishing inter-model agreement without human annotation.

Our central finding is a nuanced negative result: knowledge augmentation does not improve LLM-judged narrative quality. Despite the intuitive appeal of incorporating domain knowledge, it slightly decreases narrative quality scores for both embedding models. Meanwhile, embedding model choice turns out to have a much larger effect: SPECTER achieves notably higher mathematical coherence than MiniLM (0.94 vs. 0.81). We also observe a dissociation between mathematical coherence and LLM-judged quality, which suggests that these two metrics are capturing different aspects of narrative structure. These findings, while perhaps surprising, are a valuable empirical contribution because they show which factors actually matter for narrative extraction.

Gap and contributions. Prior narrative extraction methods for scientific corpora—most directly Narrative Trails [5], which we build on—rely on embedding similarity alone and provide no systematic evidence about whether structured knowledge actually helps. The common practice in knowledge-graph IR is to assume that richer metadata yields better results [11], yet, to our knowledge, this assumption has not been tested directly for narrative extraction with OpenAlex-style concept metadata. This paper fills that gap. Our concrete contributions are:

• The KCF framework (Section 4): This is a minimal, clearly documented modification of Narrative Trails that injects a knowledge layer (OpenAlex concept metadata) alongside embedding similarity via a single interpretable weight $\alpha$.
• An empirical negative finding (Section 6.1, Section 6.2 and Section 6.3): Knowledge augmentation does not improve LLM-judged narrative quality; it mildly degrades it. We document this with 600 paired evaluations and a five-level $\alpha$ sweep on 500 pairs.
• A dissociation between mathematical coherence and LLM-judged quality (Section 6.4 and Section 7.2): The model that maximizes mathematical coherence (SPECTER) is not the model that maximizes perceived narrative quality (MiniLM).
• Cross-model reliability evidence (Section 6.5): A total of 13 LLM judges from two providers produce highly correlated rankings (median $r=0.71$), supporting LLM-as-a-judge as a practical evaluation protocol for this task.

We treat the negative finding itself as a contribution: in a community where positive results dominate, empirically refuting a plausible hypothesis helps steer downstream effort toward more promising directions (e.g., salience-weighted knowledge and section-level signals).

Downstream application scenarios. The findings directly affect systems that build narratives over scientific corpora, such as (i) literature-review assistants that synthesize reading paths for researchers entering a new subfield, (ii) technology-surveillance pipelines that trace the evolution of a method across conferences, and (iii) curriculum generators that order readings for graduate courses. In each scenario, the choice between “knowledge-augmented” and “embedding-only” directly affects the generated path; our results indicate that the extra machinery of OpenAlex-style concept augmentation is not worth the complexity when LLM-judged narrative quality is the criterion.

The rest of this paper proceeds as follows. In Section 2 we survey related work on narrative extraction, knowledge graphs in IR, LLM-based evaluation, and scientific document embeddings. Section 3 gives a formal problem definition accompanied by a running example. Section 4 details the Knowledge-Coherence Framework, and Section 5 describes our experimental design, including all six method configurations and the 13-model evaluation protocol. We present the results in Section 6, discuss their implications and limitations in Section 7, and conclude in Section 8.

2. Related Work

2.1. Narrative Extraction and Coherence

Over the past decade, the computational extraction of narratives from document collections has moved from chronological ordering toward graph-based approaches across a range of domains [2,4,12]. Table 1 summarizes the key methods and their characteristics.

Table 1. Comparison of narrative extraction methods.

Method	Narrative Definition	Dataset	Evaluation	Knowledge Source
Connect-the-Dots [13]	Coherent chain between articles	News	Human judges	None (word activation)
Narrative Maps [14]	Graph of storylines	News	Human + coherence	None (embeddings)
Narrative Trails [5]	Max-capacity paths	News, scientific	LLM judges	None (embeddings)
KCF (ours)	Knowledge-augmented paths	Scientific literature	LLM judges (13 models)	OpenAlex metadata

Shahaf and Guestrin’s Connect-the-Dots algorithm [13] was, to our knowledge, the first to cast narrative extraction as a pathfinding problem, using linear programming to maximize the minimum coherence along a chain of news articles, relying on word activation patterns to measure inter-document similarity. Keith and Mitra [14] later moved beyond chain structures with Narrative Maps, which model narratives as graphs of multiple storylines built over document-level embeddings—we borrow their angular-and-topic-similarity coherence metric for the mathematical evaluation in the present work. More recently, German et al. [5] proposed Narrative Trails, which reframes the task as a maximum capacity path problem and achieves large speed gains without sacrificing narrative quality.

An important caveat is that narrative extraction carries different meanings depending on the domain. In the context of news and disinformation detection, narratives typically refer to repeated claims about target entities [2]. In the scientific literature—which is our focus—a narrative traces the conceptual evolution of a research topic through a sequence of documents. Because our framework and evaluation are tailored to the scientific literature, extending them to other domains with different narrative definitions remains an open question.

Human navigation studies on Wikipedia [15] reveal that people draw on both high-level conceptual connections and lexical similarity when they build paths between topics. Since different types of similarity are aspect dependent [16], purely similarity-based approaches may not fully capture the way humans connect ideas.

We note that Connect-the-Dots and Narrative Maps are not directly comparable baselines for our study because they target different narrative structures (chains vs. graphs), use different corpora, and optimize different objectives. Instead, we compare six configurations of our framework (two embedding models × three path types) to isolate the effects of embedding choice and knowledge augmentation.

A common thread runs through this line of work: the quality of a narrative path is measured by bottleneck coherence—the minimum pairwise similarity between consecutive documents. Connect-the-Dots [13] introduced this max–min formulation, Narrative Maps [14] kept it as the objective for graph-based narratives, and Narrative Trails [5] reframed it as a maximum-capacity path problem with substantial speed gains. KCF inherits bottleneck coherence from this lineage; our contribution is not a new coherence definition but a controlled test of whether augmenting that coherence with a knowledge layer helps.

The closest prior work in spirit is the recent Agenda-based Narrative Extraction method [17], which likewise modifies the Narrative Trails pipeline by injecting an external signal (LLM-derived agendas) to steer pathfinding. KCF differs in the signal used (structured OpenAlex concept metadata instead of LLM agendas) and in the research question (we test whether such augmentation actually helps on average, and find it does not).

To be concrete about why we do not baseline against these earlier systems: they output graphs of storylines rather than single paths, operate over news corpora with news-tuned embeddings, and in some cases rely on human-in-the-loop steering—so there is no apples-to-apples comparison with our single-path, scientific-literature, fully automatic setting. The single-path scope itself is inherited from Narrative Trails (validated on both news and scientific corpora); the multi-path/graph-structured extension is orthogonal and out of scope for this study.

2.2. Knowledge Graphs in Information Retrieval

The integration of knowledge graphs into IR systems has shown consistent improvements across tasks [11,18]. Wise et al. [19] developed a COVID-19 knowledge graph to address limitations of semantic-only search engines, demonstrating the value of knowledge graph augmentation for specialized domains. Ali et al. [20] showed that incorporating citation networks and topical information improves paper recommendation through heterogeneous network embeddings. Given these positive outcomes in retrieval and recommendation, a reasonable next step is to ask whether the same kind of augmentation carries over to narrative extraction—an empirical question that our experiments address directly.

However, these retrieval and recommendation settings evaluate each result independently: a KG-augmented system scores well whenever the individual returned documents are relevant. Narrative extraction is a fundamentally different problem because the unit of evaluation is the sequence: even if every intermediate document is plausibly relevant, the path as a whole can feel disjointed. Success in IR ranking therefore does not automatically translate into success in narrative extraction, and our experiments provide what is, to our knowledge, the first direct test of this assumption in the narrative setting.

For our knowledge layer we rely on OpenAlex [6], an open scholarly index that covers more than 200 million works. OpenAlex tags each work with hierarchical concepts and provides per-concept confidence scores, which makes it a practical source of structured metadata for our purposes. OpenAlex assigns concepts automatically using a machine learning classifier trained on labeled data, producing hierarchical annotations at six specificity levels (0 = most general, 5 = most specific), each with an associated confidence score between 0 and 1. Recent work has demonstrated that OpenAlex has reference coverage and metadata quality comparable to proprietary databases such as Web of Science and Scopus [21] and that its linked data representation is useful for understanding research landscapes [22].

2.3. LLM-Based Evaluation

The use of LLM-as-a-judge methods has gained traction as an alternative to expensive human evaluation [10]. Zheng et al. [23] demonstrated that GPT-4 models achieve high agreement with human judges on various NLP tasks. Keith [24] showed that multi-model LLM evaluation achieves high inter-rater reliability for narrative extraction tasks specifically.

Recent studies have raised concerns about biases in LLM evaluation, including position bias and self-enhancement bias [25,26], though frameworks such as G-Eval [27] have demonstrated that LLM-based evaluation can achieve strong human alignment. LLM-as-a-judge has also become standard practice in the narrative extraction literature specifically: Narrative Trails [5] uses it as the primary evaluation protocol, the Agenda-based Narrative Extraction work [17] relies on it to compare pathfinding variants, and [24] shows that multi-model LLM evaluation is a reliable proxy for mathematical coherence in this task. Our study adopts the same convention, which is now accepted in the IR narrative extraction community. We address LLM-evaluation bias concerns not through human evaluation but through multi-model replicated evaluation: by evaluating narratives with 13 different LLM judges from two provider families (OpenAI and Anthropic) and measuring cross-model agreement, we provide computational evidence for evaluation reliability. This approach does not replace human evaluation—which remains a limitation—but provides a practical validation of the evaluation protocol when human annotation is infeasible.

The choice of 13 judges is dictated by two considerations. First, we span two provider families (OpenAI and Anthropic) so that any systematic bias tied to a single training pipeline would show up as a reduced cross-provider correlation. Second, within each family we span multiple capability tiers (from small models such as GPT-5-nano and Claude Haiku 4.5 up to flagship models such as GPT-5.2 and Claude Opus 4.5); if small and large models agree, it becomes harder to attribute the results to a peculiarity of any single model. The exact count of 13 reflects what was available through the respective APIs at the time of the experiments.

2.4. Scientific Document Embeddings

General-purpose sentence embedding models such as MiniLM [7,8] produce representations based on semantic textual similarity. In contrast, domain-specific models such as SPECTER [9] are trained on citation relationships, learning to place documents that cite each other closer in embedding space. Because citation relationships capture intellectual influence rather than surface-level similarity, SPECTER embeddings may better reflect the conceptual connections relevant to narrative coherence. Our experiments test this hypothesis directly.

We deliberately limit the embedding comparison to MiniLM and SPECTER rather than sweeping over every recent encoder (e.g., SciNCL, SPECTER2, E5, BGE, and GTE). The pair MiniLM vs. SPECTER is the canonical general-purpose-vs. scientific-domain contrast used in Narrative Trails [5] and subsequent narrative extraction work, and keeping this pair fixed lets our results be directly compared with the existing narrative extraction literature. Exploring whether more recent encoders widen, close, or flip the mathematical–coherence-vs. LLM–quality gap is an interesting question for follow-up work but would expand the design space beyond the controlled ablation we set out to run here.

3. Problem Definition

Narrative extractionis the task of constructing an ordered sequence of documents that traces a coherent conceptual path between a given source and target document in a collection. Formally, it is as follows.

Definition 1.

Given a document collection $D=\{d_1,d_2,\dots ,d_n\}$, a source document $d_s\in D$, and a target document $d_t\in D$, the narrative extraction problem is to find an ordered sequence (path) $P=(d_s=p_1,p_2,\dots ,p_k=d_t)$ where each $p_i\in D$, such that P maximizes narrative coherence—the degree to which consecutive documents form meaningful conceptual transitions.

The quality of a path can be measured by its bottleneck coherence: the minimum pairwise coherence along the path, ${min}_{i=1}^{k-1}\theta (p_i,p_{i+1})$, where $\theta$ is a coherence function. This bottleneck formulation identifies the weakest link in the narrative chain—the point of greatest conceptual discontinuity.

The choice of bottleneck (minimum) rather than average coherence is deliberate and inherited from the established narrative extraction line of work [5,13,14]. Intuitively, a narrative is only as good as its weakest transition: a path with nine strong transitions and one jarring leap reads worse than a path with ten uniformly mediocre transitions, even though the latter may have lower average coherence. Averaging would let a single high-similarity pair compensate for an otherwise broken chain; the max–min formulation rules out this masking effect and is the standard in the field.

Running Example. Consider a collection of visualization research papers. A researcher wants to trace the conceptual path from “A Survey of Transfer Functions for Volume Rendering” (1998) to “Perception-Enhanced Visualization” (2012). A purely embedding-based path might find a short, direct route through papers with similar vocabulary (e.g., via a single intermediate paper on spectral volume rendering), yielding high mathematical coherence (min: 0.818). In contrast, a knowledge-augmented path traces a longer route through seven papers, following conceptual threads flagged by OpenAlex: volume rendering → medical visualization → opacity controls → time-varying data → perception. This longer path sacrifices 12% of mathematical coherence (min: 0.720) but may capture richer conceptual connections. The central empirical question of this paper is whether such knowledge-augmented paths actually produce better narratives.

The number of narratives extracted per source–target pair is exactly one: we extract the single optimal path according to the coherence function. The source and target documents are specified as input (see Section 5.5 for how they are sampled for evaluation).

Extracting a single path rather than a graph of multiple paths is also a deliberate design choice, not an oversight. The narrative extraction literature has explored both options: Narrative Maps [14] outputs graphs of interlinked storylines, whereas Connect-the-Dots [13] and Narrative Trails [5] output a single chain between user-specified endpoints. KCF is built directly on top of Narrative Trails, so it inherits the single-path formulation. This choice keeps the evaluation controlled (one path per endpoint pair, directly comparable across conditions) and isolates the effect of knowledge augmentation from the orthogonal question of graph structure. Extending KCF to a Narrative-Maps-style graph output is a natural follow-up direction but would require a different evaluation protocol and is out of scope for this paper.

4. The Knowledge-Coherence Framework

KCF extends traditional embedding-based narrative extraction by incorporating a parallel knowledge layer derived from scholarly metadata. The framework operates through four stages: (1) embedding-based coherence computation, (2) knowledge extraction and filtering, (3) dual-layer graph construction, and (4) weighted pathfinding.

4.1. Base Coherence Computation

Following prior work [5,14], we define the base coherence between documents $d_i$ and $d_j$ as

$${\theta }_{\mathrm{base}}(d_i,d_j)=\sqrt{S({\mathbf{z}}_i,{\mathbf{z}}_j)·T({\widehat{\mathbf{z}}}_i,{\widehat{\mathbf{z}}}_j)}$$

(1)

where:

• ${\mathbf{z}}_i,{\mathbf{z}}_j\in {\mathbb{R}}^m$ are the document embedding vectors (produced by MiniLM or SPECTER; see Section 5.2);
• $S({\mathbf{z}}_i,{\mathbf{z}}_j)=1-{\displaystyle \frac{arccos(cos({\mathbf{z}}_i,{\mathbf{z}}_j))}{\pi }}$ is the angular similarity between embeddings, normalized to $[0,1]$;
• ${\widehat{\mathbf{z}}}_i,{\widehat{\mathbf{z}}}_j\in {\mathbb{R}}^c$ are cluster membership probability vectors in a reduced space obtained via UMAP dimensionality reduction followed by HDBSCAN clustering [28], where c is the number of discovered clusters;
• $T({\widehat{\mathbf{z}}}_i,{\widehat{\mathbf{z}}}_j)=1-\mathrm{JSD}({\widehat{\mathbf{z}}}_i,{\widehat{\mathbf{z}}}_j)$ is the topic similarity based on Jensen–Shannon divergence [29] between these cluster membership distributions.

The geometric mean of angular and topic similarity captures both local semantic proximity and global topical alignment [14]. The geometric-mean form is inherited from Narrative Maps; it is preferred over an arithmetic average because it penalizes imbalance: a pair of documents that are angularly close but topically distant (or vice versa) receives a low geometric-mean score, whereas an arithmetic mean would let one strong component mask a weak one. We keep this definition unchanged for direct comparability with prior narrative extraction results.

4.2. Knowledge Extraction from OpenAlex

For each document $d_i$ in the collection, we extract concepts $C_i$ from OpenAlex metadata. Each concept $c\in C_i$ has:

• A display name (e.g., “Information visualization”, “Transfer function”);
• A confidence score $\mathrm{score}\left(c\right)\in [0,1]$, indicating how confidently OpenAlex’s classifier assigns that concept to the paper;
• A specificity level $\mathrm{level}\left(c\right)\in \{0,1,2,3,4,5\}$, where 0 is most general (e.g., “Computer Science”) and 5 is most specific (e.g., “Treemap”).

OpenAlex assigns concepts using a machine learning model trained on labeled scientific papers; confidence scores and levels are produced automatically. Of the 3549 papers in our dataset, 3527 (99.4%) had OpenAlex metadata. We exclude OpenAlex research topics and disciplinary fields, which tend to be overly broad and would create uninformative connections.

4.3. Concept Filtering Pipeline

We implement a sequential three-stage filtering pipeline to retain only informative concepts. The stages are applied in order, each building on the output of the previous:

Stage 1: Level-based filtering. We retain only concepts with specificity level $\ge {\tau }_{\ell }$, where ${\tau }_{\ell }=2$, removing generic terms like “Computer Science” that connect too many papers:

$$C_i^{\left(1\right)}=\{c\in C_i:\mathrm{level}\left(c\right)\ge {\tau }_{\ell }\}$$

(2)

Stage 2: Frequency-based filtering. Concepts appearing in more than a fraction ${\tau }_f=0.5$ (50%) of papers in the collection D are removed, as they are too common to provide discriminative connections:

$$C_i^{\left(2\right)}=\left\{c\in C_i^{\left(1\right)}:{\displaystyle \frac{\left|\right\{d_j\in D:c\in C_j\left\}\right|}{\left|D\right|}}\le {\tau }_f\right\}$$

(3)

Stage 3: Score-based filtering. For concepts at the minimum retained level ($\mathrm{level}\left(c\right)={\tau }_{\ell }$), we additionally require confidence scores below ${\tau }_s=0.95$ to avoid overly broad concepts that the classifier assigns with high confidence:

$$C_i^{\mathrm{final}}=\{c\in C_i^{\left(2\right)}:\mathrm{score}\left(c\right)<{\tau }_s\mathrm{or}\mathrm{level}\left(c\right)>{\tau }_{\ell }\}$$

(4)

We chose these thresholds (${\tau }_{\ell }=2$, ${\tau }_f=0.5$, ${\tau }_s=0.95$) based on the distribution of concepts in our dataset: level-2 concepts are the first level at which concepts become specific enough to meaningfully distinguish subfields (e.g., “Data visualization” vs. “Computer Science”), the 50% frequency threshold removes concepts shared by a majority of papers, and the 0.95 score threshold filters only very-high-confidence broad concepts at the boundary level. We stress that these values are not tuned on the downstream LLM evaluation—they are set once from the concept-frequency distribution in VisPubData before any evaluation is run. The alpha sensitivity analysis in Section 6.3 (with $\alpha =0$ fully disabling the knowledge layer and $\alpha =1$ using it alone) functions as an implicit ablation: since the knowledge component does not significantly improve LLM scores across the full range of $\alpha$ (see Section 6.3), finer threshold tuning is unlikely to reverse the negative finding.

4.4. Knowledge Similarity Computation

We measure knowledge similarity between papers by combining three components: exact concept matches, partial token overlaps, and semantic similarity of concept names.

Exact concept matching. Direct overlap between filtered concept sets, weighted by confidence scores:

$${\mathrm{Sim}}_{\mathrm{exact}}(d_i,d_j)=\sum _{c\in C_i^{\mathrm{final}}\cap C_j^{\mathrm{final}}}{\mathrm{score}}_i\left(c\right)·{\mathrm{score}}_j\left(c\right)·w_{\mathrm{exact}}$$

(5)

where an “exact” match means that both papers carry the same OpenAlex concept identifier (identical display name and identifier), and $w_{\mathrm{exact}}=1.0$ is the weight assigned to exact concept matches. When two papers share the same concept, the product of their confidence scores reflects how strongly each paper is associated with that concept.

Partial token matching. For concept pairs from different papers that are not exact matches but share meaningful tokens (tokens of length $\ge 6$ characters):

$${\mathrm{Sim}}_{\mathrm{partial}}(d_i,d_j)=\sum _{\begin{array}{c}{c}_i\in C_i^{\mathrm{final}},c_j\in C_j^{\mathrm{final}}\\ c_i\ne c_j,\mathrm{overlap}(c_i,c_j)>{\tau }_o\end{array}}{\displaystyle \frac{\left|\mathrm{common\_tokens}\right(c_i,c_j\left)\right|}{\left|\mathrm{min\_tokens}\right(c_i,c_j\left)\right|}}·w_{\mathrm{partial}}$$

(6)

where $w_{\mathrm{partial}}=0.5$, ${\tau }_o=0.5$ is the minimum token overlap ratio, and $\mathrm{common\_tokens}$ and $\mathrm{min\_tokens}$ denote the shared and minimum token sets respectively.

Semantic similarity of concept names. For concept pairs not matched by the above methods, we compute the cosine similarity between concept name embeddings using MiniLM [7,8]:

$${\mathrm{Sim}}_{\mathrm{semantic}}(d_i,d_j)=\sum _{\begin{array}{c}{c}_i\in C_i^{\mathrm{unmatched}}\\ c_j\in C_j^{\mathrm{unmatched}}\end{array}}max\left(0,cos({\mathbf{e}}_{c_i},{\mathbf{e}}_{c_j})-{\tau }_{\mathrm{sem}}\right)·w_{\mathrm{semantic}}$$

(7)

where ${\mathbf{e}}_{c_i},{\mathbf{e}}_{c_j}\in {\mathbb{R}}^{384}$ are MiniLM embedding vectors of concept display names, $w_{\mathrm{semantic}}=0.3$, ${\tau }_{\mathrm{sem}}=0.7$ is the semantic similarity threshold, and $C_i^{\mathrm{unmatched}}$ denotes concepts not already matched by exact or partial matching.

The total knowledge similarity is

$${\theta }_{\mathrm{knowledge}}(d_i,d_j)=\mathrm{normalize}\left({\mathrm{Sim}}_{\mathrm{exact}}+{\mathrm{Sim}}_{\mathrm{partial}}+{\mathrm{Sim}}_{\mathrm{semantic}}\right)$$

(8)

where normalization maps the raw sum to $[0,1]$ across all document pairs.

The three component weights ($w_{\mathrm{exact}}=1.0>w_{\mathrm{partial}}=0.5>w_{\mathrm{semantic}}=0.3$) encode a deliberate prior: an identical concept label is the strongest evidence that two papers share a topic, a token overlap between distinct concept labels is weaker evidence, and a cosine-based semantic match between concept names is the weakest. The absolute values have no effect on the final path because ${\theta }_{\mathrm{knowledge}}$ is renormalized to $[0,1]$; only the ordering of the three weights matters, and that ordering follows directly from the decreasing strength of the three evidence types.

4.5. Dual-Layer Graph Construction

We construct a sparse coherence graph $G=(D,E)$ where documents are nodes and edges exist only between pairs whose base coherence exceeds the minimum spanning tree (MST) bottleneck weight. The MST bottleneck weight is defined as the maximum edge weight on the minimum-weight path between two nodes in the MST of the complete coherence graph; it serves as a connectivity threshold ensuring that the graph remains connected while filtering weak edges [5]. MST filtering is imported unchanged from Narrative Trails: it is the tightest possible edge-pruning criterion that still guarantees connectivity, which is essential for the max–min (bottleneck) objective because any disconnected endpoint pair would yield no path at all. A looser threshold would leave many uninformative edges; a stricter one risks disconnecting valid source–target pairs.

For each edge $(d_i,d_j)\in E$, we compute a combined score:

$${\theta }_{\mathrm{combined}}(d_i,d_j)=(1-\alpha )·{\theta }_{\mathrm{base}}(d_i,d_j)+\alpha ·{\theta }_{\mathrm{knowledge}}(d_i,d_j)$$

(9)

where $\alpha \in [0,1]$ is the knowledge weight parameter controlling the balance between embedding similarity and conceptual connections. When $\alpha =0$, the framework reduces to purely embedding-based pathfinding; when $\alpha =1$, only knowledge similarity is used.

4.6. Path Extraction Algorithms

We compare three pathfinding approaches:

Knowledge-based paths. Using the combined scoring function (Equation (9)), we apply a modified Dijkstra’s algorithm with a max–min objective to find paths that maximize the minimum edge weight [5]:

$$P_{\mathrm{kb}}^{*}=arg\underset{P\in \mathcal{P}(s,t)}{max}\underset{(d_i,d_j)\in P}{min}{\theta }_{\mathrm{combined}}(d_i,d_j)$$

(10)

Base paths (no knowledge). Using only base coherence, following the Narrative Trails algorithm [5],

$$P_{\mathrm{base}}^{*}=arg\underset{P\in \mathcal{P}(s,t)}{max}\underset{(d_i,d_j)\in P}{min}{\theta }_{\mathrm{base}}(d_i,d_j)$$

(11)

Random baseline. Chronologically ordered random samples of documents between endpoints, with path length constrained between the minimum and maximum lengths of the knowledge-based and base paths for each endpoint pair, ensure comparable path lengths.

Scalability. The dominant costs are precomputable and offline. The base coherence matrix is $O\left(\right|D{|}^2)$ but is computed once; MST-bottleneck filtering is $O\left(\right|D{|}^{2}log\left|D\right|)$ using Kruskal’s algorithm, after which the resulting coherence graph is much sparser than the complete graph. OpenAlex concept metadata and concept-name MiniLM embeddings are fetched and computed once and cached. Each online pathfinding query then reduces to a modified Dijkstra with max–min updates on the sparse graph, i.e., $O\left(\right|E|+|D|log|D\left|\right)$ per endpoint pair. On the 3549-paper VisPubData collection, a single path extraction completes in well under a second on commodity hardware, and the full 600-pair main experiment is dominated by LLM-evaluation latency, not pathfinding.

5. Experimental Design

5.1. Dataset: VisPubData

We evaluate KCF on the VisPubData collection [30], containing metadata and abstracts for papers from major IEEE visualization conferences. After filtering papers without abstracts, our dataset comprises 3549 papers from 1990 to 2022, covering over three decades of visualization research. The collection encompasses 172 automatically discovered topics via HDBSCAN clustering [28]. We use paper titles and abstracts for embedding generation and LLM evaluation. OpenAlex metadata was available for 99.4% of papers. Raw data used in the experiments is included in the Supplementary Material of this paper for ease of access.

We deliberately restrict evaluation to a single domain for three reasons. First, VisPubData is a well-established benchmark in the visualization and narrative extraction communities, already used in prior work including Narrative Trails [5], which makes our results directly comparable to that literature. Second, OpenAlex provides near-complete (99.4%) metadata coverage on this corpus, so the knowledge layer can be evaluated on equal footing for essentially every document; extending to domains with sparser metadata would confound the negative finding with missing-data effects. Third, the study is designed as a controlled ablation of knowledge augmentation: a single, well-understood corpus is the appropriate test bed for isolating the effect of $\alpha$, embedding model, and knowledge layer without introducing between-domain variance. Generalization to other scientific subfields (e.g., biomedical or social-science literature) is discussed in Section 7.6 as future work.

5.2. Embedding Models

We compare two embedding paradigms:

• MiniLM (all-MiniLM-L6-v2) [7,8]: A general-purpose sentence transformer producing 384-dimensional embeddings, trained on semantic textual similarity tasks. It captures surface-level semantic similarity.
• SPECTER [9]: A SciBERT-based model producing 768-dimensional embeddings, trained on citation triplets (paper, cited paper, non-cited paper). It captures document-level relatedness based on citation patterns, which may better reflect intellectual influence.

5.3. Method Configurations

We evaluate six method configurations formed by crossing two embedding models with three path types:

1.

Random (MiniLM): Random paths in MiniLM embedding space.

2.

Random (SPECTER): Random paths in SPECTER embedding space.

3.

MiniLM: Base coherence paths using MiniLM embeddings ($\alpha =0$).

4.

SPECTER: Base coherence paths using SPECTER embeddings ($\alpha =0$).

5.

MiniLM + K: Knowledge-augmented paths using MiniLM embeddings ($\alpha =0.5$).

6.

SPECTER + K: Knowledge-augmented paths using SPECTER embeddings ($\alpha =0.5$).

For knowledge-augmented configurations, we set $\alpha =0.5$ to equally balance embedding similarity and knowledge; the alpha sensitivity analysis (Section 6.3) evaluates this choice.

5.4. Evaluation Metrics

Average mathematical coherence. The mean of pairwise base coherence values ${\theta }_{\mathrm{base}}(p_i,p_{i+1})$ along a path is computed using the embedding model corresponding to each configuration. This measures the overall semantic smoothness of the narrative. We use “mathematical coherence” throughout to distinguish this model-computed, automatic quantity from the LLM-judged “narrative quality” described next; the term is inherited from the Narrative Maps/Narrative Trails line of work and is distinct from human-perceived coherence, which is not directly measured in this study.

LLM-judged coherence. We follow the evaluation protocol of Keith [24] and use GPT-4o-mini [31] as the primary judge with a two-tier scoring protocol matched to each experiment’s analytical purpose. To mitigate noise from any single API call, we query the model $K=3$ times per narrative, each time with a different random seed, and report the mean of the three scores. We refer to this procedure as scoring independence.

Direct scoring is used for the alpha sensitivity analysis (Section 6.3), where the goal is to detect relative trends across a five-level parameter sweep (2500 total evaluations). Each narrative is rendered as a numbered list of paper titles and abstracts, and the model is prompted: “Rate the coherence of this narrative sequence on a scale of 1–10… Provide a single coherence score (1–10) based on how well the articles flow together.” This lightweight protocol is well suited to exploratory analyses in which relative ranking across configurations matters more than absolute calibration.

Structured scoring is used for the main six-method comparison (Section 6.1), which serves as the confirmatory experiment. Here, the prompt enforces a chain-of-thought evaluation: the judge must (i) analyze each consecutive transition for specific methodological or topical connections, (ii) count the number of weak links that lack a clear connection, and (iii) assess whether the sequence exhibits logical progression before assigning a final 1–10 score with explicit calibration guidelines. In practice, this structured protocol produces more discriminative scores because it penalizes sequences whose papers merely share a broad domain without exhibiting specific inter-paper connections.

Because the two protocols differ in prompt complexity and calibration strictness, their absolute score scales are not directly comparable. However, within each experiment all methods are evaluated under identical conditions, so relative comparisons within a table remain valid.

Why LLM-as-a-judge rather than human evaluation. Our evaluation relies exclusively on LLM judges, which is a deliberate design choice grounded in accepted community practice. LLM-as-a-judge has become the standard evaluation protocol in the narrative extraction IR community: it is the primary evaluation in Narrative Trails [5], in Agenda-based Narrative Extraction [17], and has been explicitly validated as a reliable proxy for mathematical coherence in narrative extraction by [24]. Human annotation of hundreds of multi-document scientific narratives by domain experts is resource intensive and would meaningfully limit sample size; we instead spend the evaluation budget on redundancy (three seeds per narrative) and cross-model replication (13 judges). We acknowledge the absence of human evaluation as a limitation (Section 7.6) but argue that the 13-model agreement result provides meaningful computational evidence in its place.

Prompt design and bias mitigation. Each prompt presents the narrative as a numbered list of “Paper i: TITLE—ABSTRACT” entries, followed by the scoring instruction. We apply three bias mitigations inherited from the LLM-as-a-judge literature [23,24,25,27]: (i) repeated sampling with three different random seeds per narrative, averaged to reduce single-call noise; (ii) cross-model replication across 13 judges from two providers, which reveals any systematic bias tied to a single model or family; and (iii) identical presentation across conditions, so that configurations differ only in which documents appear (not in how they are rendered), ruling out presentation confounds. We do not randomize the order of the documents because the paths are temporally ordered and the LLM is told this is a narrative; position within the sequence is part of the stimulus, not a confound. Calibration between the two scoring protocols (direct vs. structured) is handled by keeping each experiment internally consistent and never comparing absolute scores across protocols.

5.5. Paired-Samples Design

Our design is within-subjects: each source–target endpoint pair is evaluated across all method configurations so that comparisons are not confounded by endpoint-specific characteristics. In the main six-method comparison we use 600 evaluation data points (100 endpoint pairs × 6 methods), and in the alpha sensitivity analysis we use 500 pairs across 5 alpha values.

Each endpoint pair satisfies: (1) temporal separation of at least two years, (2) different topic clusters (ensuring non-trivial paths), and (3) valid connectivity in the coherence graph. All path types for a given endpoint pair receive identical knowledge augmentation treatment through inline annotations showing concept connections, ensuring that LLMs evaluate narrative quality based on actual conceptual connections, not presentation differences.

5.6. Multi-Model LLM Evaluation Protocol

As a check on whether our LLM-based scores reflect genuine narrative quality or merely the idiosyncrasies of a single model, we repeat the evaluation with 13 different judges drawn from two provider families:

OpenAI models: GPT-4o-mini, GPT-4o, GPT-5-nano, GPT-5-mini, GPT-5, GPT-5.1, and GPT-5.2.

Anthropic models: Claude Opus 4.1, Claude Haiku 4.5, Claude Sonnet 4.5, Claude Opus 4.5, Claude Sonnet 4, and Claude Opus 4.

Every model sees the same narratives and the same prompt. We then compute pairwise Pearson correlations across all $\left({\displaystyle \genfrac{}{}{0pt}{}{13}{2}}\right)=78$ model pairs; if models from different providers and different capability tiers largely agree, we take that as reasonable computational evidence for evaluation reliability.

5.7. Statistical Analysis Methods

For our statistical analysis, we use the Friedman test [32] as a non-parametric omnibus test to detect differences across related groups. When significant effects are found, we follow up with pairwise Wilcoxon signed-rank tests [33] and apply Bonferroni correction [34] to keep the family-wise error rate at 0.05. We also compute Cohen’s d [35] for paired samples as a standardized effect size, interpreting values as small ($\left|d\right|\approx 0.2$), medium ($\left|d\right|\approx 0.5$), or large ($\left|d\right|\ge 0.8$).

6. Results

6.1. Main Results: Six-Method Comparison

Table 2 presents our main results across all six method configurations, evaluated on 600 data points (100 endpoint pairs × 6 methods) with the structured scoring protocol (see Section 5.4). Because this protocol applies stricter calibration than the direct scoring used in the alpha sensitivity analysis, the absolute LLM scores in Table 2 and Table 3 should not be compared directly; however, relative comparisons within each table remain valid.

Table 2. Main results: comparison of six narrative extraction configurations. Values show mean ± standard deviation across 100 endpoint pairs. Bold indicates best in column.

Method	Avg Coherence	LLM Score
Random (MiniLM)	$0.56\pm 0.11$	$3.62\pm 0.83$
Random (SPECTER)	$0.87\pm 0.05$	$3.68\pm 0.85$
MiniLM	$0.81\pm 0.04$	$\mathbf{5}.\mathbf{87}\pm \mathbf{0}.\mathbf{82}$
SPECTER	$\mathbf{0}.\mathbf{94}\pm \mathbf{0}.\mathbf{02}$	$5.36\pm 0.92$
MiniLM + K	$0.78\pm 0.03$	$5.25\pm 0.95$
SPECTER + K	$0.91\pm 0.02$	$4.66\pm 0.87$

Table 3. Alpha sensitivity analysis: effect of knowledge weight $\alpha$ on coherence and LLM scores (MiniLM embeddings, 500 pairs). Values show mean ± standard deviation.

$\mathit{\alpha }$	Path Length	Min Coherence	Avg Coherence	LLM Score
0.0	$18.5\pm 17.5$	$0.758\pm 0.087$	$0.815\pm 0.038$	$7.83\pm 0.54$
0.3	$12.5\pm 8.0$	$0.722\pm 0.079$	$0.799\pm 0.031$	$7.90\pm 0.52$
0.5	$8.5\pm 3.9$	$0.688\pm 0.079$	$0.776\pm 0.033$	$7.91\pm 0.51$
0.7	$7.1\pm 2.4$	$0.640\pm 0.076$	$0.746\pm 0.032$	$7.81\pm 0.60$
1.0	$6.3\pm 2.4$	$0.583\pm 0.069$	$0.685\pm 0.047$	$7.57\pm 0.71$

Figure 1 visualizes the method comparison, showing both average coherence and LLM scores. Reading the figure, the left panel shows that the two SPECTER-based structured methods (SPECTER and SPECTER + K) dominate on mathematical coherence, while the right panel shows that the two MiniLM-based structured methods lead on LLM score—a clear visual signature of the dissociation discussed in Section 7.2.

Figure 1. Comparison of six narrative extraction method configurations. Left: average mathematical coherence. Right: LLM-judged coherence scores.

Three observations stand out.

Structured vs. random. The gap between any structured method and either random baseline is large and unambiguous ($p<0.001$ in every comparison, with large effect sizes). This is expected but confirms that the structured methods behave as intended.

Embedding model choice has a pronounced effect. SPECTER yields 0.94 average coherence versus 0.81 for MiniLM, a 16% gap. Notably, even the Random (SPECTER) baseline achieves 0.87, which exceeds the structured MiniLM paths (0.81). The citation-informed training behind SPECTER produces a tighter embedding space in which even randomly sampled documents tend to be more coherent. This high Random (SPECTER) score is initially surprising but has a concrete explanation: SPECTER is trained on citation triplets, which pulls all papers from a scientifically coherent subfield close together in embedding space, so randomly chosen papers from VisPubData—itself a topically focused corpus—already sit in a small neighborhood of that space. Notably, however, the same baseline receives a low LLM score (3.68, similar to Random (MiniLM) at 3.62): neighborhood tightness in embedding space does not translate into a readable narrative, which reinforces the dissociation we discuss in Section 7.2.

Knowledge augmentation hurts rather than helps. Contrary to our initial hypothesis, adding knowledge decreases LLM scores for both embedding models: MiniLM drops from 5.87 to 5.25 ($-10.6\%$), and SPECTER drops from 5.36 to 4.66 ($-13.1\%$). We return to possible explanations in Section 7.1.

6.2. Effect of Knowledge Augmentation

The negative effect of knowledge augmentation on LLM scores is statistically significant for both embedding models. Table 4 presents key pairwise comparisons with Wilcoxon signed-rank tests and Bonferroni correction.

Table 4. Key pairwise comparisons (Wilcoxon signed-rank test with Bonferroni correction). $\Delta$ Mean refers to average coherence. The check mark symbol (✔) indicates whether the comparison is statistically significant.

Comparison	$\mathbf{\Delta }$ Mean	p-Value	Cohen’s d	Sig.
Random (MiniLM) vs. SPECTER + K	$-0.346$	<0.0001	$-3.23$	✔
MiniLM vs. MiniLM + K	$+0.039$	<0.0001	$+1.11$	✔
SPECTER vs. SPECTER + K	$+0.029$	<0.0001	$+2.37$	✔
MiniLM + K vs. SPECTER + K	$-0.132$	<0.0001	$-3.89$	✔

The effect of knowledge augmentation on mathematical coherence is consistently negative: MiniLM coherence drops from 0.81 to 0.78 (Cohen’s $d=1.11$, large) and SPECTER from 0.94 to 0.91 (Cohen’s $d=2.37$, large). This is expected—incorporating knowledge diverts paths from the mathematically optimal embedding-based route. However, the parallel decrease in LLM scores is the more noteworthy finding, as knowledge augmentation was hypothesized to improve perceived narrative quality.

6.3. Alpha Sensitivity Analysis

To further investigate knowledge augmentation, we conducted a sensitivity analysis across five values of $\alpha \in \{0.0,0.3,0.5,0.7,1.0\}$ with 500 test pairs using MiniLM embeddings and the direct scoring protocol (see Section 5.4), which is appropriate for detecting relative trends across a multi-level parameter sweep. Table 3 and Figure 2 present the results.

Figure 2. Alpha sensitivity analysis. Left: average coherence decreases steadily with $\alpha$. Right: LLM scores remain essentially flat for $\alpha \in [0.0,0.7]$, then drop at $\alpha =1.0$ (pure knowledge).

Two trends stand out. On the one hand, mathematical coherence drops steadily as $\alpha$ grows: it starts at 0.815 when $\alpha =0.0$ and falls to 0.685 at $\alpha =1.0$—a 16% decline. On the other hand, LLM scores barely move over the range $\alpha \in [0.0,0.7]$, fluctuating between 7.81 and 7.91, and only dip noticeably at $\alpha =1.0$ (where the score falls to 7.57, i.e., when the system relies on knowledge alone). In short, while moderate values of $\alpha$ produce marginally higher point estimates than $\alpha =0$ (e.g., 7.91 vs. 7.83), these differences are small relative to the standard deviations and no value of $\alpha$ significantly outperforms the purely embedding-based configuration, confirming that knowledge augmentation is at best negligibly beneficial and at worst harmful for perceived narrative coherence.

An ancillary observation is that path length shrinks monotonically with $\alpha$—from a mean of 18.5 documents when $\alpha =0.0$ down to 6.3 when $\alpha =1.0$—because the knowledge-weighted edges tend to create shortcuts that bypass the longer, embedding-optimal routes.

The practical implication of this flat LLM–quality curve is that knowledge augmentation is essentially “free from above but not beneficial”: it does not destroy narrative quality in the moderate range $\alpha \in [0.3,0.7]$, but it also does not produce any reliable gain relative to pure embedding-based coherence (i.e., $\alpha =0$). Combined with the mathematical–coherence decline, the cost–benefit balance favors $\alpha =0$, i.e., not augmenting at all. Follow-up work that hopes to extract gains from knowledge augmentation will therefore need to change the form of the knowledge signal rather than the mixing weight.

6.4. Embedding Model Comparison: SPECTER vs. MiniLM

The effect of embedding model choice is large and consistent. SPECTER produces paths with substantially higher mathematical coherence than MiniLM across all configurations:

• Random paths: SPECTER 0.87 vs. MiniLM 0.56 ($+55\%$).
• Base paths: SPECTER 0.94 vs. MiniLM 0.81 ($+16\%$).
• Knowledge-augmented paths: SPECTER + K 0.91 vs. MiniLM + K 0.78 ($+17\%$).

This advantage of SPECTER in mathematical coherence likely reflects its citation-informed training: papers that cite each other are placed closer in embedding space, creating a tighter and more structured space for pathfinding.

However, SPECTER does not achieve higher LLM scores. MiniLM paths (5.87) score higher than SPECTER paths (5.36) on LLM-judged quality, and MiniLM + K (5.25) scores higher than SPECTER + K (4.66). This reveals a dissociation between mathematical coherence and LLM-judged quality: the embedding model producing the most mathematically coherent paths does not produce the paths that LLM judges rate most highly.

6.5. LLM Evaluation Reliability

Table 5 presents the cross-model Pearson correlation matrix for 13 LLM judges. Figure 3 visualizes the agreement structure.

Table 5. Cross-model agreement: Pearson correlations among 13 LLM judges. Values above 0.70 are in bold. The median off-diagonal correlation is 0.71.

	GPT-4o-m	GPT-4o	GPT-5-n	GPT-5-m	GPT-5	Cl.Op.4.1	Cl.Ha.4.5	Cl.So.4.5	Cl.Op.4.5	GPT-5.1	GPT-5.2	Cl.So.4	Cl.Op.4
GPT-4o-mini	1.00	0.79	0.72	0.72	0.50	0.55	0.78	0.78	0.54	0.70	0.80	0.60	0.60
GPT-4o	0.79	1.00	0.61	0.76	0.70	0.82	0.88	0.90	0.82	0.81	0.86	0.79	0.87
GPT-5-nano	0.72	0.61	1.00	0.56	0.34	0.46	0.64	0.62	0.28	0.52	0.52	0.54	0.43
GPT-5-mini	0.72	0.76	0.56	1.00	0.67	0.56	0.84	0.80	0.71	0.83	0.84	0.48	0.57
GPT-5	0.50	0.70	0.34	0.67	1.00	0.71	0.80	0.66	0.81	0.76	0.75	0.53	0.71
Cl. Opus 4.1	0.55	0.82	0.46	0.56	0.71	1.00	0.77	0.82	0.68	0.61	0.68	0.89	0.98
Cl. Haiku 4.5	0.78	0.88	0.64	0.84	0.80	0.77	1.00	0.88	0.78	0.83	0.92	0.71	0.79
Cl. Sonnet 4.5	0.78	0.90	0.62	0.80	0.66	0.82	0.88	1.00	0.64	0.68	0.82	0.87	0.86
Cl. Opus 4.5	0.54	0.82	0.28	0.71	0.81	0.68	0.78	0.64	1.00	0.90	0.84	0.45	0.72
GPT-5.1	0.70	0.81	0.52	0.83	0.76	0.61	0.83	0.68	0.90	1.00	0.87	0.44	0.62
GPT-5.2	0.80	0.86	0.52	0.84	0.75	0.68	0.92	0.82	0.84	0.87	1.00	0.56	0.70
Cl. Sonnet 4	0.60	0.79	0.54	0.48	0.53	0.89	0.71	0.87	0.45	0.44	0.56	1.00	0.89
Cl. Opus 4	0.60	0.87	0.43	0.57	0.71	0.98	0.79	0.86	0.72	0.62	0.70	0.89	1.00

Figure 3. Cross-model agreement heatmap for 13 LLM judges. Warmer colors indicate higher Pearson correlation.

The median off-diagonal Pearson correlation comes out to 0.71, with the majority of pairs exceeding $r=0.60$. As one would expect, the highest agreement levels appear between models from similar families and capability tiers (e.g., Claude Opus 4.1 and Claude Opus 4: $r=0.98$; Claude Haiku 4.5 and GPT-5.2: $r=0.92$; and GPT-4o and Claude Sonnet 4.5: $r=0.90$). The lowest agreements involve GPT-5-nano ($r=0.28$ with Claude Opus 4.5), the smallest model in our pool.

At a practical level, these correlations tell us that our LLM-based evaluation is not an artifact of any single model’s scoring tendencies: judges from two separate provider families, spanning a wide capability range, arrive at broadly concordant rankings. We do not claim this as a methodological novelty—prior work has already established LLM-as-a-judge reliability in other settings [23,24]—but the result does validate our evaluation protocol for the specific task of assessing narrative coherence.

6.6. Qualitative Analysis

Returning to the running example from Section 3, we can illustrate the typical pattern. A base MiniLM path from “Transfer Functions for Volume Rendering” to “Perception-Enhanced Visualization” finds a direct three-step route through spectrally similar papers (min coherence: 0.818, LLM score: 4.33). The knowledge-augmented path, by contrast, traces a seven-step route through medical visualization, opacity controls, and time-varying data (min coherence: 0.720, LLM score: 6.33).

Although this particular example shows a higher LLM score for the knowledge path, it does not represent the aggregate pattern (where knowledge augmentation decreases LLM scores on average). It does illustrate that knowledge augmentation can sometimes uncover valuable conceptual bridges, but on average these detours do not improve the perceived narrative quality. The aggregate statistics (Table 2 and Table 4) provide the reliable picture.

7. Discussion

7.1. Why Knowledge Augmentation Does Not Help

Why does adding structured knowledge actually lower narrative quality scores? We can think of at least three plausible explanations, though the current data do not allow us to fully disentangle them.

The likeliest explanation is a mismatch between the granularity of OpenAlex concept tags and the kind of conceptual continuity that makes a narrative feel coherent. Two papers may both carry the tags “Volume rendering” and “Medical imaging” and therefore score high on knowledge similarity, yet the research questions they actually address could be quite different—one might focus on GPU acceleration, the other on surgical planning. In other words, sharing a concept label is not the same as sharing a research thread, and the LLM judges appear to pick up on this distinction.

Furthermore, we note that the alpha sensitivity results support this reading: at $\alpha =1.0$, paths driven by knowledge alone score 7.57 on narrative quality—well below the embedding-based baseline of 7.83 ($\alpha =0.0$), but far above the random-path baseline of 3.62–3.68. The knowledge signal carries genuine domain information; the problem is resolution.

A related issue is that knowledge-weighted paths tend to route through papers connected by concept overlap rather than by tight topical focus. The result is what one might call “detour narratives”: sequences that touch many related concepts but lack the steady, incremental progression that readers—and, it appears, LLM judges—associate with coherent storytelling. Embedding-based paths, by contrast, already capture intellectual proximity quite effectively (especially SPECTER, whose citation-informed training encodes real influence relationships between papers).

In particular, the path-length data in Table 3 make this concrete: mean path length falls from 18.5 documents at $\alpha =0.0$ to 6.3 at $\alpha =1.0$. Knowledge augmentation creates shorter shortcuts, not richer detours—and LLM judges reward step-by-step conceptual progression, not topical co-occurrence.

Finally, our knowledge similarity function treats every shared concept equally, regardless of whether a concept is central to a paper’s contribution or merely tangential. Weighting concepts by their salience to each paper’s core argument—for instance, using the concept’s position in the abstract or its frequency relative to the paper’s own topic distribution—might produce a more informative knowledge signal. More specifically, the three mechanisms point to a common source: the knowledge signal operates at a granularity suited to topic-level retrieval, not to the thread-level continuity that narrative extraction requires. We leave this refinement for future work.

7.2. The Dissociation Between Mathematical and LLM-Judged Coherence

One of the more notable results in our study is the gap between mathematical coherence and what LLM judges actually prefer. SPECTER paths score 0.94 on mathematical coherence—higher than MiniLM’s 0.81—yet LLM judges rate MiniLM narratives more favorably (5.87 vs. 5.36). The data indicate that these two metrics capture quite different aspects of narrative quality.

Recall that mathematical coherence, as we define it, quantifies how close consecutive documents sit in embedding space (via angular and topic similarity). A high score on this metric simply means that neighboring papers are nearby in the representation. LLM judges, however, seem to reward something different: whether the sequence goes somewhere—whether it develops a line of thought, broadens scope, or builds toward a synthesis. Because SPECTER’s citation-informed training produces a tightly clustered embedding space, consecutive papers along a SPECTER path can be extremely similar to one another, and the resulting narrative may feel repetitive or “stuck in place” despite being mathematically smooth.

This dissociation has implications beyond our narrative extraction setting. A persistent pattern in IR and NLP evaluation is to use embedding-based similarity (or embedding-based retrieval metrics) as a stand-in for user-perceived quality. Our data caution against this substitution whenever the downstream unit of consumption is a sequence rather than an individual item: the embedding representation most favorable for clustering or retrieval (SPECTER here) is not necessarily the representation most favorable for sequence-level narrative reading. Analogous dissociations are known in other sequence-level NLP tasks—e.g., BLEU vs. human judgment in machine translation, or ROUGE vs. human judgment in summarization—and our results add narrative extraction to that list. Practitioners should therefore pick the metric (and the embedding model) that matches the use case: mathematical coherence when the goal is automatic downstream processing, LLM-/human-judged quality when the goal is reader-facing narrative output.

7.3. Embedding Model Selection for Narrative Extraction

From a practitioner’s standpoint, the takeaway is that embedding model choice is not a minor implementation detail—it shapes the character of the extracted narratives in fundamental ways. When the goal is to maximize mathematical coherence (e.g., for downstream clustering or automated analysis), SPECTER is the clear winner; its citation-based training produces paths whose consecutive documents are tightly related. When, instead, the goal is to produce narratives that a human reader (or, as a proxy, an LLM judge) would rate as high-quality, MiniLM seems to be the better option, possibly because its broader notion of semantic similarity yields paths with more conceptual progression and less redundancy.

For researchers designing narrative extraction pipelines, the implication is clear: the definition of “good narrative” in a given application context should drive the choice of embedding model. The two models we tested optimize for different facets of narrative structure, and conflating the two objectives risks producing misleading conclusions.

Concrete takeaways for future work. Our results suggest three concrete directions. First, change the form of the knowledge signal: rather than computing pairwise knowledge similarity from shared concept labels, future work could weight each concept by its salience to the paper’s core contribution (e.g., via position in the abstract, attention over section headers, or LLM-extracted paper-level claims). Second, move beyond single paths: the Narrative Maps formulation outputs graphs of multiple storylines and could be extended in a similar manner to KCF. Third, align evaluation with use case: when the downstream consumer is a human reader, LLM-as-judge (optionally paired with small-scale human studies for future validation) should be the primary target; when the consumer is an automated downstream system, mathematical coherence remains appropriate.

7.4. Reliability of LLM-Based Evaluation

With a median pairwise correlation of $r=0.71$ across 13 judges, the overall agreement is high—especially among the larger models. GPT-4o, Claude Sonnet 4.5, and Claude Haiku 4.5, for instance, all correlate with each other above $r=0.85$. The main outlier is GPT-5-nano, the smallest model in our pool, whose scores diverge more from the rest.

We note that this analysis builds on an established paradigm—LLM-as-a-judge has been validated in multiple settings [10,23]—rather than introducing a new evaluation methodology. Its contribution here is domain-specific: it demonstrates that LLM judges can score narrative coherence in scientific literature with strong consistency, a finding that supports both the present study and future work in this area where human annotation may be unavailable.

One distinction is worth making explicit: cross-model agreement measures rank consistency among LLM judges, not alignment with human assessors. The structural feature that makes our result meaningful is that the 13 judges span two organizationally independent providers—OpenAI and Anthropic—with different training corpora, reinforcement-learning procedures, and alignment objectives; correlated errors that cross this boundary are harder to attribute to a single shared artifact than agreement within one provider family. Meta-analyses of LLM-as-a-judge report human–LLM score correlations broadly in the 0.7–0.9 range across quality-assessment tasks [10], placing our cross-model median $r=0.71$ in the regime where LLM rankings are known to track human assessment. A targeted human pilot study remains the appropriate direct validation, as noted in the Section 7.6.

7.5. LLM-Generated Narratives as Future Work

One might ask: if LLMs can judge narratives, why not use them to generate narratives directly? This approach is conceptually distinct from our graph-based pathfinding framework. LLM-generated narratives would not be constrained to actual documents in the collection and would raise different questions about faithfulness and grounding. We consider LLM-based narrative generation an important direction for future work, but it addresses a different formulation of the narrative extraction problem.

7.6. Limitations

Domain scope. Our evaluation is restricted to scientific literature (specifically visualization research) where OpenAlex provides structured metadata. The framework could potentially be applied to other domains where knowledge resources exist (news archives with entity annotations, legal repositories with precedent relationships), but empirical validation in these domains remains future work. The negative finding rests on a mechanism—concept-label granularity being too coarse for narrative-thread discrimination—that is not specific to visualization research; it reflects how knowledge graphs are typically constructed, where annotations capture topic membership rather than argumentative continuity. The directional result (concept-level augmentation is at best neutral) may therefore extend to other scientific corpora with similar annotation density, though specific effect sizes will depend on the precision of available metadata.

No human evaluation. We rely entirely on LLM judges, validated through cross-model agreement. While 13-model agreement provides computational evidence for reliability, human evaluation remains the gold standard for assessing narrative quality. As discussed in Section 7.4, the cross-provider structure of our judge pool reduces the risk of correlated biases, but that is a structural argument rather than direct evidence of human alignment. The absence of human evaluation is an important limitation of this study, and—as discussed in Section 5.4 and Section 7.4—reflects a deliberate design choice aligned with accepted practice in the IR narrative extraction community; we flag a targeted human pilot study as the natural next step.

Title and abstract evaluation. Narratives are presented to LLM judges as sequences of paper titles and abstracts. Richer representations (full text, figures, conclusions) might enable more nuanced evaluation but would also increase cost and complexity.

Knowledge source specificity. Our knowledge augmentation uses OpenAlex concept annotations, which are automatically generated and may not capture all relevant conceptual relationships. Different knowledge sources (citation networks, co-authorship graphs, ontologies) might yield different results.

Fixed framework parameters. While we vary $\alpha$ systematically, other parameters (filtering thresholds, similarity weights) are held fixed. A more comprehensive parameter search might reveal better configurations, though the alpha sensitivity analysis suggests that the knowledge component itself is the limiting factor.

8. Conclusions

We set out to answer a deceptively simple question: does augmenting embedding-based narrative extraction with structured domain knowledge produce better narratives? To find out, we built the Knowledge-Coherence Framework (KCF), integrated OpenAlex concept metadata into the pathfinding process, and ran experiments along three axes—embedding model choice (MiniLM vs. SPECTER), knowledge augmentation (with and without, plus a five-point alpha sweep), and evaluation reliability (13 LLM judges from two provider families).

The answer, perhaps surprisingly, is negative. Knowledge augmentation via OpenAlex concept metadata does not raise LLM-judged narrative quality; if anything, it lowers scores slightly for both embedding models. The factor that does make a large difference is the embedding model itself, and it does so in a way that is central to our message: SPECTER achieves notably higher mathematical coherence than MiniLM (0.94 vs. 0.81), yet MiniLM narratives receive higher marks from LLM judges (5.87 vs. 5.36). The model that wins on the automatic metric loses on the LLM-judged metric. This dissociation between mathematical coherence and perceptual narrative quality is, in our view, one of the most important outcomes of the study: it shows that in narrative extraction the two metrics are not interchangeable optimization targets, and that practitioners should pick the metric that matches the intended consumer (automated pipeline vs. human reader) before selecting an embedding model. The 13-model cross-evaluation (median $r=0.71$) gives us reasonable confidence that these patterns are not artifacts of a single judge’s biases.

These findings suggest that for narrative extraction in scientific literature: (1) embedding model selection matters more than knowledge augmentation, (2) mathematical coherence and perceived narrative quality are not equivalent optimization targets, and (3) future work on knowledge augmentation should focus on more targeted representations of conceptual relevance rather than broad concept overlap.

Future work could investigate whether different knowledge representations (citation paths, author networks, section-level semantic analysis) yield different results, extend evaluation to other scientific domains and non-scientific corpora, and incorporate human evaluation to complement LLM-based assessment.