Mind the Link: Discourse Link-Aware Hallucination Detection in Summarization

Abstract

Recent studies on detecting hallucinations in summaries follow a method of decomposing summaries into atomic content units (ACUs) and then determining whether each unit logically matches the document text based on natural language inference. However, this fails to consider discourse link relations such as temporal order, causality, and purpose, leading to the inability to detect conflicts in semantic connections between individual summary ACUs, even when the conflicts are present in the document. To overcome this limitation, this study proposes a method of extracting Discourse Link-Aware Content Unit (DL-ACU) by converting the summary into an Abstract Meaning Representation (AMR) graph and structuring the discourse link relations between ACUs. Additionally, to align summary ACUs with corresponding document information in a fine-grained manner, we propose a Selective Document-Atomic Content Unit (SD-ACU). For each summary ACU, the SD-ACU retrieves only the most relevant document sentences and then decomposes them into document ACUs. Applying the DL-ACU module to existing hallucination detection systems such as FIZZ and FENICE reduces the error rate of discourse link errors on FRANK. When both modules are combined, the system improves balanced accuracy and ROC-AUC across major benchmarks. This suggests the proposed method effectively captures discourse link errors while enabling ACU-to-ACU alignment.

1. Introduction

Recently, in NLI-based hallucination detection in summarization systems, summaries are divided into Atomic Content Units (ACUs) [1,2,3,4] and then each fact is verified for consistency with the original document. However, such systems do not consider discourse link information [5] such as temporal order, causality, and purpose between summary ACUs, resulting in discourse link errors (LinkE). In this study, we define the concept of a discourse link not only as relations between sentences but also as a more comprehensive concept that includes semantic connections between ACUs. We observe that even though each ACU is individually in an entailment relationship with sentences in the document, false positive predictions occur when the temporal order, causality, or purpose between them does not align with the document. For example, consider a document stating that “Pfizer announced the results of its phase 3 trial on Monday…The FDA granted emergency use authorization three days later”. A summary that incorrectly claims “The FDA granted emergency use authorization before Pfizer announced its trial results” would still appear consistent when its individual ACUs (“The FDA granted emergency use authorization” and “Pfizer announced its trial results”) are evaluated separately. However, our Discourse Link-Aware Content Unit, which preserves the temporal relation (“The FDA granted authorization before Pfizer announced its results”), correctly identifies the error. This error does not stem from a limitation of the NLI model. Rather, it arises because the input ACUs do not capture discourse link information. Explicitly modeling these links between ACUs is essential for reliable factuality, not only verifying isolated units. Prior work underscores this importance. FRANK [5] formalizes discourse link error and quantifies its prevalence, while FactFT [6] and AMRFact [7] inject temporal and causal perturbations to train stronger evaluators.

In hallucination detection for summarization, a common approach is to decompose the document and the summary and then assess whether document chunks match summary chunks [2,3,4,8,9]. Concretely, consistency is assessed by pairing a summary unit with a candidate document unit and feeding the pair to an NLI model to obtain an entailment decision. However, most prior work extracts ACUs only from the summary while keeping the document at the sentence level. Applying ACU extraction to only one side induces a granularity mismatch that manifests in two cases. First, when the summary is decomposed into ACUs but the document remains at the sentence level, a single document sentence may contain multiple events or facts, making it difficult to find a unit that exactly corresponds to an atomic statement in the summary. Second, an alternative line of research [10] has focused on decomposing the premise, but without a corresponding decomposition of the hypothesis, which also leads to a granularity mismatch. Therefore, a truly precise evaluation requires decomposing both the summary and the document to an atomic level to enable ACU-to-ACU alignment; while ideal, decomposing the entire document to the same extent as the summary would dramatically increase the computational cost of ACU generation models [11,12] and the total number of NLI comparisons required.

To address these two challenges, we propose two complementary modules. An overview of the two pipelines is shown in Figure 1. First, we introduce a new unit called the Discourse Link-Aware Content Unit (DL-ACU), which captures the discourse link relation between ACUs. This unit is constructed by converting summary sentences into AMR graphs [13] and extracting subgraphs centered on discourse link relation markers such as time, cause, and purpose. These subgraphs contain discourse link information between different ACUs, which is then converted to text and filtered using NLI-based filtering to select only high-quality DL-ACUs. DL-ACU can judge the entailment relationship with the document while preserving discourse link information within the summary. Second, we propose the Selective Document-Atomic Content Unit (SD-ACU) to address granularity mismatch and computational cost. For each summary ACU, we retrieve the top-k most relevant document sentences using an entailment score and decompose only those sentences into document-side ACUs. This selective decomposition filters out extraneous information (like attributions or modifiers) from the document sentence, enabling a more precise ACU-to-ACU alignment. For instance, a summary ACU like “the rebels attacked civilians” might initially appear to have weak support when compared against a full document sentence cluttered with attribution, such as “Human-rights monitors stated that rebel groups shot at bystanders…”. By decomposing this source sentence, our method allows for a direct comparison with the relevant document-side ACU, “rebel groups shot at bystanders”, which provides a much clearer and more accurate entailment match. This strategy not only improves ACU-to-ACU alignment but also reduces the number of NLI calls by approximately 30% compared to fully decomposing the document in the FENICE baseline, and reduces Language Model (LM) calls for decomposition by 83% on average in the AggreFact-FtSota benchmark, all while improving balanced accuracy over sentence-level premises. As early quantitative evidence, our best variant improves average balanced accuracy on AggreFact-FtSota by 1.39 percentage points over FIZZ and by 0.99 points over FENICE; on SummaC the gains are 1.78 and 1.36 points, respectively. On DiverSumm, ROC-AUC increases by 0.79 and 2.66 points for FIZZ and FENICE, and on FaithBench by 2.07 and 1.95 points.

Consistent with these gains, the error analysis shows that DL-ACU reduces the LinkE rate by 6.81 percentage points for FIZZ and by 11.36 points for FENICE relative to their family baselines.

2. Related Works

2.1. Hallucination Detection in Summarization

Recent Large Language Models (LLMs) demonstrate outstanding performance in summary generation, but they still seriously suffer from hallucination problems, which involve including information that does not exist in the document. This means that the summary sentences are not consistent with the document. Approaches to evaluating the factuality of summaries can be broadly categorized into NLI-based, QA-based, and LLM-based methodologies.

NLI-based methodologies treat the summary as a hypothesis and the document as a premise to evaluate factuality. They generally follow one of two paradigms. The first, and more common, paradigm involves decomposing the summary and document and using a pre-trained NLI model to verify the resulting chunks [2,3,4,8,9]. The second paradigm focuses on synthesizing negative (hallucinated) samples to fine-tune a specialized NLI model for the task [6,7,14,15]. Systems following the first paradigm, such as SummaC-ZS [8], AlignScore [9], InFusE [4], FIZZ [2], and FENICE [3], typically follow a three-step procedure, the general flow of which is illustrated on the left side of Figure 1. This process involves (1) decomposing the document and summary into chunks; (2) for each summary chunk, calculating NLI scores against all document chunks and selecting the highest score as the evidence; and (3) merging the scores for each summary chunk using an average (soft aggregation) or minimum (hard aggregation) to compute a final score. For instance, SummaC-ZS, AlignScore, InFusE, and FENICE use soft aggregation, while FIZZ uses hard aggregation, while effective, these methodologies uniformly decompose the document only at the sentence level, creating a granularity mismatch when the summary is broken into finer-grained ACUs. By contrast, the second paradigm, including models like FactCC [15], FactFT [6], and AMRFact [7], synthesizes hallucinated summaries as negative samples to train NLI models. In particular, FactFT and AMRFact curate negatives that instantiate challenging error types like LinkE and CorefE [5]. However, because these approaches do not decompose the inputs, they offer limited fine-grained error analysis and can suffer from input truncation on long-document benchmarks like DiverSumm [4] and FaithBench [16].

Beyond NLI, QA-based methods (e.g., QuestEval [17], QAFactEval [18]) and LLM-based evaluators (e.g., G-Eval [19], ACUEval [20]) have also been explored. Graph-based methods like FactGraph [21] evaluate semantic overlap using AMR graphs, while DAE [22] performs arc level entailment over dependency graphs.

While many methods exist, prior work typically omits discourse link relations and compares fine-grained summaries with sentence-level documents, causing a granularity mismatch. Addressing these two issues can improve evidence selection and reduce computation in NLI-based verification, especially for long documents.

2.2. Atomic Content Unit Decomposition

Atomic Content Unit (ACU) is a verifiable unit of information that cannot be further subdivided, and regarded as the minimal semantic unit for precise factuality assessment of summaries [1,23]. Leveraging ACUs enables localized detection of non-factual spans within a summary; subsequent DAE [22] demonstrated that such decomposition effectively identifies which part of the generation is non-factual.

Early approaches primarily operated at the sentence level. SummaC [8] and AlignScore [9] decompose summaries into sentences. More recent studies pursue finer evaluation by extracting summaries at the ACU level. For instance, InFusE [4], FENICE [3] and FIZZ [2] employ large generative models. T5-base [12], Orca-2 [11] extract ACUs from summaries, following the definition proposed in ROSE [1]. ROSE defines an ACU as the smallest verifiable unit; FENICE describes an ACU as an elementary information unit found in the summary, and FIZZ characterizes an ACU as a short and concise information unit. Fundamentally, the definitions from FIZZ and FENICE align with that of ROSE, and both studies use the ROSE benchmark to evaluate the ACU decomposition capabilities of their respective models, T5-base_FENICE and ORCA-2_FIZZ.

2.3. AMR Graph

Abstract Meaning Representation (AMR) is a predicate-centered semantic graph formalism that was introduced in early work [13]. It abstracts sentence meaning to capture deep semantic structure. AMR graphs are commonly rendered in PENMAN notation [24] and encode relations among predicates, core roles (:ARG0, :ARG1, etc.), and non-core roles (:time, :location, etc.). For instance, “He studied to pass the exam” expresses purpose via :purpose marker, while “Alice gave a present to Bob on Monday” makes action, participants, and time explicit via :ARG0, :ARG1, :ARG2, :time markers. Such representations are advantageous for modeling semantic structures that simple surface cues cannot capture. The performance of sequence-to-graph (STOG) and graph-to-sequence (GTOS) models has improved dramatically with the advent of Transformer-based architectures [12,25], enabling their use in a wide range of downstream AMR applications [26].

Our ACU decomposition follows a recent baseline [27]. Baseline first parses each sentence into an AMR graph using a sequence-to-graph model. From the graph, it selects subgraphs that connect a predicate with its core roles. Each subgraph is turned into an ACU by the graph-to-sequence model.

We extend this pipeline to capture discourse link information. We form subgraphs that make the links explicit, such as temporal order (:time), causality (:cause), and purpose (:purpose). From these, we extract the Discourse Link-Aware Content Unit (DL-ACU). This helps reveal discourse link errors that recent ACU-based verifications [2,3,4] often miss.

3. Methods

As illustrated in Figure 1, the right panel (Steps 1–4) shows the Discourse Link-Aware Content Unit (DL-ACU) module, and the left panel (Steps 5 and 6) shows the Selective Document-ACU (SD-ACU) module. Step 7 performs NLI scoring and aggregation shared by both families. We explicitly reference these numbered steps below to align the text with the figure.

3.1. Discourse Link-Aware Content Unit Decomposition Using AMR Graphs

When a summary S is given, we aim to obtain the set F of ACUs contained in it. In particular, our objective is to account for discourse links that prior ACU decomposition [2,3,4] methods have overlooked. Our method consists of five steps. An overview of this process is visualized on the right side of Figure 1.

3.1.1. Subgraph Extraction Based on Discourse Link

As shown in Step 1 of Figure 1, we first build an AMR graph for the entire summary using a sequence-to-graph (STOG) model.

$$\mathcal{G}=\mathtt{STOG}\left(S\right)$$

Let this AMR be a labeled directed graph $\mathcal{G}=(V,E,\mathsf{\ell })$. In this structure, V is the set of vertices (or nodes), which represent the core concepts and entities in the summary (e.g., actions like attack-01 or entities like rebel). E is the set of edges, representing all the connections between the nodes in V. An edge is a triple that defines a directed link: (source_node, relation_label, target_node); for instance, (attack-01, :ARG0, rebel) is one complete edge in the set E. Finally, ℓ is the labeling function, which assigns a name to each node, such that $\mathsf{\ell }\left(v\right)$ is the node label.

We define the set of predicate nodes (e.g., impose-01, attack, capture).

$$\mathcal{P}=\{v\in V\mid \mathtt{isPredicate}(\mathsf{\ell }\left(v\right)\left)\right\}$$

and the set of discourse link relations.

$${\mathcal{L}}_{\mathrm{dl}}=\{:\mathtt{cause},:\mathtt{purpose},:\mathtt{time}\}$$

In Step 2 of Figure 1, we extract discourse link triples between predicates.

$$\mathcal{T}=\{(h,r,t)\in E\mid h,t\in \mathcal{P},r\in {\mathcal{L}}_{\mathrm{dl}}\}$$

In Figure 1; for example, $(\mathtt{impose}-\mathtt{01},:\mathtt{cause},\mathtt{attack})$ and $(\mathtt{attack},:\mathtt{time},\mathtt{capture})$ are such triples. For each $\tau =(h,r,t)\in \mathcal{T}$, we build a subgraph by keeping the discourse link information and expanding the local predicate–argument structure around its endpoints.

$${\mathcal{G}}_{\tau }=\left\{(h,r,t)\right\}\cup \mathtt{ARG}\_\mathtt{NEIGH}\left(h\right)\cup \mathtt{ARG}\_\mathtt{NEIGH}\left(t\right)$$

where the one-hop argument neighborhood is defined as the set of outgoing edges from a node whose relations are in a predefined set $\mathcal{A}$.

$$\mathtt{ARG}\_\mathtt{NEIGH}\left(v\right)=\left\{\right(v,a,u)\in E\mid a\in \mathcal{A}\}$$

Here, the rule set $\mathcal{A}$ is configured to retain core semantic roles:

$$\mathcal{A}=\left\{\begin{array}{cc}& :\mathtt{ARG}\mathtt{0},:\mathtt{ARG}\mathtt{1},:\mathtt{ARG}\mathtt{2},:\mathtt{ARG}\mathtt{3},:\mathtt{ARG}\mathtt{4},:\mathtt{ARG}\mathtt{5}\hfill \end{array}\right\}$$

We exclude name, metadata and list-internal edges via

$${\mathcal{L}}_{\mathrm{skip}}=\{:\mathtt{wiki},:\mathtt{name},:\mathtt{op}\}$$

:ARGx capture “who did what to whom,” which designate the core roles of an event and thus are preserved as essential semantics. In contrast, :wiki, :name and :op are representational metadata that do not affect propositional content and introduce noise. This yields, for the discourse link triple $(\mathtt{impose}-\mathtt{01},:\mathtt{cause},\mathtt{attack})$, a subgraph that retains impose-01 with $:\mathtt{ARG}\mathtt{0}=$ government and $:\mathtt{ARG}\mathtt{1}=$ sanction, the discourse link triple $(\mathtt{impose}-\mathtt{01},:\mathtt{cause},\mathtt{attack})$ itself, and attack with $:\mathtt{ARG}\mathtt{0}=$ rebel and $:\mathtt{ARG}\mathtt{1}=$ civilian, matching Figure 1 subgraphs.

3.1.2. Post-Processing

In Step 3 of Figure 1, each subgraph ${\mathcal{G}}_{\tau }$ is first transformed into a candidate atomic content unit text ${\widehat{a}}_t$ using a graph-to-sequence (GTOS) model.

$${\widehat{a}}_t=\mathtt{GTOS}\left({\mathcal{G}}_{\tau }\right)$$

In Step 4 of Figure 1, the resulting candidate texts ${\widehat{a}}_t$ undergo a three-step post-processing pipeline to ensure quality and relevance. First, a Coreference Resolution step is applied, where pronouns or other referential expressions in ${\widehat{a}}_t$ are resolved in the context of the full summary S to obtain the final expression $a_t$. Second, to ensure factual consistency with the summary, we apply NLI-based Factuality Filtering, keeping only those units $a_t$ whose entailment score meets a threshold $\tau$.

$${\mathcal{F}}_{\mathrm{DL}}=\{a_t\mid f_{\mathrm{NLI}\_\mathrm{ENTAIL}}(\mathrm{premise}=S,\mathrm{hypothesis}=a_t)\ge \tau \}$$

Finally, an Overlap Filtering and Merging step is performed to avoid redundancy with the ACUs already produced by a baseline system (${\mathcal{F}}_{\mathrm{base}}$). The new DL-ACUs are merged only if their BERTScore similarity to any existing baseline ACU is below a threshold $\delta$.

$$\mathcal{F}={\mathcal{F}}_{\mathrm{base}}\cup \{a_t\in {\mathcal{F}}_{\mathrm{DL}}\mid \forall a_b\in {\mathcal{F}}_{\mathrm{base}},\mathtt{BERTScore}(a_t,a_b)<\delta \}$$

For completeness, we summarize the entire DL-ACU pipeline in Algorithm 1, aligning the procedural flow with Figure 1 and making the subgraph extraction and post-processing reproducible.

Algorithm 1 DL-ACU: Discourse Link-Aware ACU Decomposition

Require: Summary S; STOG; GTOS; NLI scorer fNLI_ENTAIL; coreference resolver Coref; thresholds τ, δ

Require: AMR role sets: predicate detector isPredicate(·); discourse labels ${\mathcal{L}}_{\mathrm{dl}}=\{:\mathtt{cause},:\mathtt{purpose},:\mathtt{time}\}$; rule set $\mathcal{A}$; skip set ${\mathcal{L}}_{\mathrm{skip}}$ (:wiki, :name, :op)

Require: Baseline ACUs ${\mathcal{F}}_{\mathrm{base}}$ (from the host Hallucination Detection system)

1: $\mathcal{G}=(V,E,\mathsf{\ell })\leftarrow \mathtt{STOG}\left(S\right)$   2: $\mathcal{P}\leftarrow \{v\in V\mid \mathtt{isPredicate}(\mathsf{\ell }\left(v\right)\left)\right\}$   3: $\mathcal{T}\leftarrow \{(h,r,t)\in E\mid h,t\in \mathcal{P},r\in {\mathcal{L}}_{\mathrm{dl}}\}$   4: functionARG_NEIGH(v)   5:     $\mathcal{N}\leftarrow Ø$   6:     for all $(v,a,u)\in E$ do   7:         if $a\in \mathcal{A}$ and $a\notin {\mathcal{L}}_{\mathrm{skip}}$ then   8:            $\mathcal{N}\leftarrow \mathcal{N}\cup \left\{\right(v,a,u\left)\right\}$   9:         end if 10:     end for 11:     return $\mathcal{N}$ 12: end function 13: $\widehat{\mathcal{F}}\leftarrow Ø$ 14: for all  ${\tau }^{\prime }=(h,r,t)\in \mathcal{T}$do 15:     ${\mathcal{G}}_{{\tau }^{\prime }}\leftarrow \left\{(h,r,t)\right\}\cup ARG\_NEIGH\left(h\right)\cup ARG\_NEIGH\left(t\right)$ 16:     $\widehat{a}\leftarrow \mathtt{GTOS}\left({\mathcal{G}}_{{\tau }^{\prime }}\right)$ 17:     $a\leftarrow Coref(\widehat{a},\mathrm{context}=S)$ 18:     if $f_{\mathrm{NLI}\_\mathrm{ENTAIL}}(\mathrm{premise}=S,\mathrm{hypothesis}=a)\ge \tau$ then 19:         $\widehat{\mathcal{F}}\leftarrow \widehat{\mathcal{F}}\cup \left\{a\right\}$ 20:     end if 21: end for 22: ${\mathcal{F}}_{\mathrm{DL}}\leftarrow \{a\in \widehat{\mathcal{F}}\mid \forall a_b\in {\mathcal{F}}_{\mathrm{base}},\mathtt{BERTScore}(a,a_b)<\delta \}$ 23: return  $\mathcal{F}\leftarrow {\mathcal{F}}_{\mathrm{base}}\cup {\mathcal{F}}_{\mathrm{DL}}$

3.1.3. Discussion and Practical Considerations

Through this process, we can extract information that explicitly reflects the discourse link between facts in addition to existing atomic content unit information. This is thanks to the structural characteristics of the AMR graph. For example, time expressions such as ‘before’ and ‘after’ are recognized as :time, and ambiguous expressions such as ‘to’ are recognized as :purpose, allowing for more accurate decomposition of semantic connections than simple text-matching-based techniques.

Several practical considerations are also addressed in our pipeline. The generated text expressions often include pronouns or demonstrative expressions, which must be appropriately replaced with proper nouns; in this study, we use the Maverick coreference resolution model [28]. Furthermore, when the performance of the STOG or GTOS models is low, the generated ACUs may be logically inconsistent, a correctness issue mentioned in WICE [29]. To prevent such errors, we filter the extracted ACUs in our post-processing step (Section 3.1.2) based on an NLI model to ensure they are entailed in the original summary. Additionally, to avoid redundancy with DL-ACUs that might already have been generated by a baseline system, we perform a BERTScore-based [30] similarity calculation and filter out duplicates. This is particularly important for soft aggregation systems like FENICE [3], whereas hard aggregation systems like FIZZ are more robust to such duplication.

3.2. Selective Document-Atomic Content Unit Decomposition

When verifying summaries, existing NLI-based hallucination detection systems [2,3,4] decompose summaries into ACUs for verification, but documents are usually compared only at the sentence level. This makes it difficult to achieve fine-grained alignment. This study decomposes sentences in the document into ACUs, enabling fine-grained alignment between the ACUs in the summary and document. Our method consists of three steps. An overview of this process is shown on the left side of Figure 1.

3.2.1. Entailment-Based Selection

Let the input document and summary be

$$\mathcal{D}=\{d_1,d_2,\dots ,d_n\},\mathcal{S}=\{s_1,s_2,\dots ,s_m\}$$

We first obtain summary-side atomic content units using an LM [11,12] (as in the FIZZ and FENICE baselines).

$$\mathcal{A}=\bigcup _{k=1}^m\mathtt{LM}\left(s_k\right)=\{a_1,a_2,\dots ,a_t\}$$

As shown in Figure 1, we perform NLI scoring between document sentences and summary ACUs. For each summary atomic content unit $a_i\in \mathcal{A}$ and each document sentence $d_j\in \mathcal{D}$, we compute an entailment probability using an NLI model by treating $d_j$ as the premise and $a_i$ as the hypothesis.

$$\mathrm{Ent}(d_j,a_i)=f_{\mathrm{NLI}\_\mathrm{ENTAIL}}\left(premise=d_j,hypothesis=a_i\right)\in [0,1]$$

These initial entailment scores are cached for reuse in the NLI scoring phase. We then select the top-k sentences with the highest scores.

$${\mathcal{D}}_i^{\prime }=\mathtt{Top}-k\left(\{d_j\in \mathcal{D}\mid \mathrm{Ent}(d_j,a_i)\}\right)$$

3.2.2. Atomic Content Unit Decomposition

In Step 5 of Figure 1, for each selected sentence $d_u\in {\mathcal{D}}_i^{\prime }$, we extract document side atomic content units with the language model (LM). In Step 6 of Figure 1, we construct the Document-Atomic Content Unit set.

$${\mathcal{D}}_{\mathrm{chunked}}=\bigcup _{i=1}^t\{\mathtt{LM}\left(d_u\right)\mid d_u\in {\mathcal{D}}_i^{\prime }\}$$

Finally, in Step 7 of Figure 1, we perform NLI scoring and Aggregation between Document-Atomic Content Unit set and Summary Atomic Content Unit set following the baseline systems (hard aggregation in FIZZ; soft aggregation in FENICE).

3.2.3. Discussion and Practical Considerations

This method enables ACU-to-ACU alignment at the document level, allowing for precise comparison with the summary. In particular, it improves interpretability by identifying the facts in the document that are the basis for errors in the summary. Furthermore, decomposing all sentences in the entire document unnecessarily increases computational cost. Therefore, rather than decomposing the entire document, we selectively decompose only the document sentences most relevant for verifying each summary ACU. This reduces the cost of generating ACU with LM and the cost of calculating the entailment score between document chunks and summary chunks with the NLI model.

4. Experiments

4.1. Experimental Setup

We evaluate two modules for NLI-based hallucination detection in summarization: a Discourse Link-Aware Content Unit module and a Selective Document-Atomic Content Unit module. To assess the contribution of each component, the modules are integrated into recent baselines, FIZZ [2] and FENICE [3], both individually and in combination. All experiments are run on a single NVIDIA GeForce RTX 3090 GPU, and the implementation is publicly available at https://github.com/leadawon/DLSD-ACU (accessed on 6 September 2025). For the Discourse Link-Aware Content Unit module, AMR parsing (sequence-to-graph, STOG) and generation (graph-to-sequence, GTOS) rely on publicly available models at https://github.com/bjascob/amrlib-models (accessed on 15 March 2025) and for entailment scoring on two modules, we adopt the same off-the-shelf NLI model [31] used by FIZZ, namely tals/albert-xlarge-vitaminc-mnli.

We report balanced accuracy and ROC-AUC depending on the benchmark. For FRANK [5] in the SummaC benchmark [8], we additionally measure the discourse link error rate following prior work FactFT [6], which enables a focused assessment of discourse link error. This metric is calculated as the proportion of false positives within a given error category.

$$\mathrm{Error}\mathrm{Rate}=FP/N,$$

where $FP$ is the number of false-positive predictions and N is the total number of examples belonging to that specific error category. Lower values indicate better performance.

In our main experiments, we set the NLI filtering threshold to $\tau =0.4$ to decide whether a summary ACU is entailed by the original text, and we apply BERTScore-based overlap removal with a threshold $\delta =0.7$ to reduce semantic redundancy. For top-k retrieval on the document side, we use $K=1$, selecting the single most relevant sentence per summary ACU before document side ACU decomposition. We find that performance to be relatively insensitive to small variations around these values. Concretely, sweeping $\tau$ by $\pm 0.2$ around $0.4$, $\delta$ by $\pm 0.1$ around $0.7$, and K between 1 and 3 yields only marginal changes in accuracy. For coreference resolution, we use sapienzanlp/maverick-mes-ontonotes [28] (author-reported OntoNotes 5.0 test CoNLL F1 [32] 83.6). we adopt the released weights and tokenizer, without task specific fine tuning.

4.2. Evaluation Benchmarks

AggreFact [33] integrates nine existing datasets and consists of summary and document sentence pairs with human-annotated binary labels. The included sources are FRANK [5], SummEval [34], CLIFF [35], XSumFaith [36], Polytope [37], FactCC [15], Goyal’21 [38], Cao’22 [39], and Wang’20 [40]. Here, OLD denotes old summarizers such as Pointer-Generator [41] and BottomUp [42] models, EXFORMER denotes early Transformer-based summarizers (e.g., BERTSum [43] or GPT-2 [44] style systems), and FTSOTA denotes fine-tuned state-of-the-art pretrained Transformer summarizers (e.g., BART [25], PEGAsus [45], T5 [12]). The benchmark is organized by the family of the summarization model that produced the summaries (OLD, EXFORMER, FTSOTA) and by domain (CNN/DM [46], XSum [47]). CNN/DM typically supports more extractive, multi-sentence summaries, whereas XSum is highly abstractive and usually single-sentence. We focus on the AggreFact-FtSota subset, which reflects recent systems that generate fluent text while often exhibiting subtle and hard-to-detect factual hallucinations. For AggreFact-FtSota, we tune thresholds on the validation split and report balanced accuracy on the test split.

The SummaC [8] benchmark aggregates six datasets for factuality evaluation, each providing human binary labels for summary and document pairs. The constituent datasets are FactCC [15], FRANK [5], XSumFaith [36], CoGenSumm [48], Polytope [37], and SummEval [34]. As with AggreFact-FtSota, we use the validation split for threshold tuning and report balanced accuracy on the test split. For the FRANK subset, we additionally measure the LinkE error rate [6]. FRANK defines several error categories used in our analysis. LinkE occurs when a summary misstates the relation between otherwise correct facts, such as reversing event order or implying a false cause–effect. CorefE arises when referential expressions point to the wrong entity or are too ambiguous to resolve. GramE covers sentences whose grammar is so broken that factuality cannot be assessed. EntE denotes errors where the action is correct but the participants or their roles are wrong. CircE involves incorrect supporting details such as time, place, or manner. RelE indicates a mismatch in the core predicate or event. OutE refers to content introduced by the summary with no support in the source document.

DiverSumm [4] includes ChemSumm [49], ArXiv [50], GovReport [51], MultiNews [52], and QMSUM [53], covering scientific articles, meeting transcripts, government reports, and multi-document news. These datasets represent a mix of summarization styles; ChemSumm, QMSUM, and ArXiv feature primarily abstractive summaries, whereas GovReport and MultiNews contain more extractive summaries. Documents and summaries in this benchmark are also longer than in prior benchmarks.

FaithBench [16] targets hallucinations produced by recent LLM-based summarizers. It provides fine-grained hallucination labels created by human experts for summaries generated by ten modern LLMs. The dataset addresses the limitations of older benchmarks by focusing on challenging cases. Each summary span is annotated as consistent, benign, questionable, or unwanted. We use the official binary version released by the authors for direct evaluation. Because FaithBench does not include a validation split, we report ROC-AUC.

5. Results

Table 1. Balanced accuracy (%) on SummaC subsets. AVG is the average of the six datasets. Bold denotes the best score in each column.

Method	CoGenSumm	FactCC	FRANK	SummEval	Polytope	XSumFaith	AVG
Other comparators
DAE	63.40	75.90	61.70	70.30	62.80	50.80	64.15
FactCC	65.40	77.30	59.40	58.53	58.20	56.54	62.56
SummaC-ZS	70.40	83.81	78.51	78.70	62.00	58.40	71.97
SummaC-Conv	64.70	89.06	81.62	81.70	62.70	65.44	74.20
MENLI	55.20	58.17	75.11	52.73	57.07	67.15	60.91
MFMA	64.13	84.88	80.62	75.50	58.31	55.07	69.75
AlignScore	70.16	85.88	82.06	56.42	60.64	73.44	71.43
InFusE	75.61	87.59	79.39	73.54	54.44	67.64	73.04
FIZZ family
FIZZ (baseline)	59.18	78.66	80.44	62.92	58.09	72.33	68.60
FIZZ + DL-ACU	62.43	79.46	80.49	63.09	58.47	72.72	69.44
FIZZ + SD-ACU	63.23	81.50	80.70	64.17	60.14	72.55	70.38
FIZZ + DL-ACU + SD-ACU	63.45	81.52	80.68	63.35	58.89	72.97	70.14
FENICE family
FENICE (baseline)	76.50	84.29	83.05	72.32	65.48	72.90	75.76
FENICE + DL-ACU	76.43	84.29	82.34	72.29	65.49	72.62	75.58
FENICE + SD-ACU	78.02	85.33	83.28	76.66	66.79	72.66	77.12
FENICE + DL-ACU + SD-ACU	77.87	84.98	83.33	74.89	66.24	73.83	76.86

,

Table 2. Error Rate (%, smaller is better) by error category. Rows labeled $\Delta$ report the difference from the family baseline, computed as (baseline − method) in percentage points (pp); thus, positive $\Delta$ indicates a reduction in error rate (better), while negative indicates an increase (worse).

Method	LinkE	CorefE	GramE	EntE	CircE	RelE	OutE	OtherE
FIZZ family
FIZZ (baseline)	45.45	70.75	34.69	30.10	21.79	15.13	5.35	75.00
FIZZ + DL-ACU	38.64	66.98	33.33	25.37	20.51	14.47	4.71	60.00
$\Delta$ vs. baseline (pp)	6.81	3.77	1.36	4.73	1.28	0.66	0.64	15.00
FENICE family
FENICE (baseline)	90.91	94.34	72.11	65.92	55.13	49.34	32.33	90.00
FENICE + DL-ACU	79.55	93.40	73.47	68.16	60.26	49.34	34.48	90.00
$\Delta$ vs. baseline (pp)	11.36	0.94	−1.36	−2.24	−5.13	0	−2.15	0

,

Table 3. Balanced accuracy (%) on AggreFact-FtSota subsets. AVG is the average of the two columns. Bold denotes the best score in each column.

Method	AggreFact-Cnn-FtSota	AggreFact-XSum-FtSota	AVG
Other comparators
DAE	59.40	73.10	66.25
FactCC	57.60	54.88	56.24
SummaC-ZS	65.19	54.08	59.64
SummaC-Conv	61.72	63.52	62.62
MENLI	62.24	65.30	63.77
MFMA	61.86	55.00	58.43
AlignScore	62.72	69.44	66.08
InFusE	64.51	65.82	65.16
FIZZ family
FIZZ (baseline)	65.86	69.25	67.56
FIZZ + DL-ACU	67.11	69.82	68.47
FIZZ + SD-ACU	66.46	69.44	67.95
FIZZ + DL-ACU + SD-ACU	67.71	70.18	68.95
FENICE family
FENICE (baseline)	66.23	73.83	70.03
FENICE + DL-ACU	66.29	74.08	70.19
FENICE + SD-ACU	68.43	73.44	70.94
FENICE + DL-ACU + SD-ACU	67.17	74.86	71.02

,

Table 4. ROC-AUC (%) on DiverSumm subsets. CSM, MNW, QMS, AXV, and GOV refer to ChemSum, MultiNews, QMSUM, ArXiv, and GovReport, respectively. AVG is the average of the the five datasets. Bold denotes the best score in each column.

Method	CSM	MNW	QMS	AXV	GOV	AVG
Other comparators
FactCC	50.36	34.41	46.62	61.87	66.97	52.05
SummaC-ZS	59.36	46.72	44.92	68.16	72.58	58.75
SummaC-Conv	53.76	52.70	49.44	61.50	71.13	57.71
MENLI	53.19	60.53	44.45	66.85	34.39	51.48
MFMA	60.34	49.94	43.33	72.30	60.32	57.65
AlignScore	58.59	42.69	59.21	72.77	85.25	63.30
InFusE	46.04	40.05	46.71	70.49	78.19	56.70
FIZZ family
FIZZ (baseline)	54.01	38.67	51.22	62.15	64.16	54.44
FIZZ + DL-ACU	53.34	38.20	49.34	61.34	64.52	53.34
FIZZ + SD-ACU	54.27	39.82	51.79	62.81	66.20	54.98
FIZZ + DL-ACU + SD-ACU	54.89	40.16	52.44	62.62	66.02	55.23
FENICE family
FENICE (baseline)	54.83	43.50	55.92	75.94	74.34	60.91
FENICE + DL-ACU	52.60	43.96	56.11	76.13	70.23	59.41
FENICE + SD-ACU	51.57	47.18	50.00	73.83	77.47	63.57
FENICE + DL-ACU + SD-ACU	50.60	45.45	52.16	72.27	75.84	59.26

and

Table 5. ROC-AUC (%) on FaithBench. Bold denotes the best score.

Method	FaithBench
Other comparators
FactCC	49.54
SummaC-ZS	47.52
SummaC-Conv	52.16
MENLI	49.78
MFMA	52.09
AlignScore	48.33
InFusE	49.87
FIZZ family
FIZZ (baseline)	52.50
FIZZ + DL-ACU	53.32
FIZZ + SD-ACU	54.27
FIZZ + DL-ACU + SD-ACU	54.57
FENICE family
FENICE (baseline)	59.05
FENICE + DL-ACU	60.20
FENICE + SD-ACU	60.81
FENICE + DL-ACU + SD-ACU	61.00

present the performance of our proposed modules, DL-ACU and SD-ACU, when integrated into the FIZZ [2] and FENICE [3] baselines. The results indicate that our modules provide targeted improvements, with each showing distinct strengths on different types of benchmarks and error categories. The following subsections detail the specific contributions of each module.

5.1. Discourse Link-Aware Content Unit

We observe that DL-ACU performs well on its intended error type. As Table 2 shows the integration of DL-ACU significantly reduces the error rate for discourse link errors (LinkE) on FRANK, with reductions of 6.81 percentage points for FIZZ and 11.36 percentage points for FENICE. This reduction is substantially greater compared to other error types, confirming that our AMR-based approach successfully captures discourse link errors that standard ACU methods [2,3,4] often overlook. Interestingly, while the LinkE error rate decreases notably, the overall improvement in balanced accuracy on the FRANK dataset improves only slightly, by 0.05 percentage points in FIZZ, and decreases by 0.71 percentage points in FENICE (Table 1). This outcome is expected, as the DL-ACU module is specifically designed to address discourse link error (LinkE) and does not inherently target other prevalent error types.

Beyond its primary target, the DL-ACU module’s broader impact varies depending on the baseline. When applied to FIZZ, it reduces the error rate across nearly all error categories shown in Table 2, suggesting a general improvement. In contrast, with the stronger FENICE baseline, while the improvement on LinkE is even more pronounced (a reduction of 11.36 percentage points), we observe an increase in the error rate of 5.13 percentage points in Circumstance errors and 2.24 points in Entity errors when applied to FENICE. We interpret this mainly as an aggregation effect. FIZZ uses hard aggregation based on the minimum, so one most diagnostic ACU tends to decide the label, and extra DL-ACUs rarely hurt entity or circumstance cases. FENICE uses soft aggregation based on the mean, so DL-ACUs that are only partially entailed or close to neutral still enter the average and can nudge borderline entity or circumstance instances toward false positives.

Table 6. Examples from FRANK in SummaC. Baselines show ACUs produced separately by FIZZ (paired with ORCA) and FENICE (paired with T5-base). Our method yields a single discourse link-aware content unit that preserves temporal order, causality, and purpose with discourse markers highlighted in red.

ID	FRANK Summary	FIZZORCA-2	FENICET5-base	Our Method (DL-ACU)
1	However, the shooter tried to ram the gates before firing at the guard at least once.	- A shooter was involved. - The shooter tried to ram the gates. - The shooter fired at the guard. - The guard was fired at least once.	- The shooter tried to ram the gates. - The shooter fired at the guard at least once.	- The shooter rammed the gates before firing at the guard.
2	Police believe the shooter barricaded himself inside after noticing a couple fighting in a car.	- Police believe the shooter barricaded himself inside. - Shooter noticed a couple fighting in a car.	- Police believe the shooter barricaded himself inside. - The shooter noticed a couple fighting in a car.	- Police believe the shooter barricaded himself inside after noticing a couple fighting in a car.
3	French scientists say they have found a way to hide the earth’s vast mountains of Mont Blanc.	- French scientists have found a way. - The way is to hide the earth’s vast mountains. - The earth’s vast mountains are called Mont Blanc.	- French scientists found a way to hide the earth’s vast mountains of Mont Blanc	- French scientists found a way in order to hide the earth’s vast mountains of Mont Blanc.
4	The banks in the Indian capital, Delhi, have been shut down because of corruption and corruption, the BBC has learned.	- Banks in Delhi have been shut down. - The reason for the shut down is corruption. - The source of the information is the BBC.	- The banks in the Indian capital, Delhi, have been shut down because of corruption.	- The banks in Delhi were shut down because of corruption, the BBC has learned.

provides a qualitative comparison of how different decomposition models handle summaries containing explicit discourse link. These summaries from the FRANK dataset have markers like before, after, to, and because of. Such markers signify that for accurate factuality assessment, it is not enough to verify individual ACUs in isolation; the semantic connection between those ACUs must also be validated. In Examples 1 and 2, we observe that the baseline decomposers used by FIZZ and FENICE fail to capture discourse link relations, instead decomposing the summary into ACUs and losing the discourse link information. Our DL-ACU approach, in contrast, preserves the temporal order information. In Example 3, ORCA-2_FIZZ’s attempt to capture the purpose results in the awkward phrase (“the way is to”), which is not a well-formed atomic content unit, as the referent of “the way” is not specified. Many of the extracted ACUs also give the impression of being naively segmented triplets rather than genuinely considering discourse link relations. In contrast, T5-base_FENICE successfully captures the purpose relation by producing a fluent and coherent DL-ACU in this example. These examples also highlight secondary effects of our post-processing pipeline. During AMR graph construction and graph-to-sequence generation, entities are occasionally replaced by pronouns such as ‘he’ or ‘they’, potentially introducing referential ambiguity. However, as Example 2 shows, our Coreference Resolution module (see Section 3.1.2) successfully restores the original mentions (‘police’ and ‘shooter’). Finally, in Examples 3 and 4, all methods manage to correctly extract discourse link-aware content units; we observe that our Overlap Filtering module (see Section 3.1.2) effectively eliminates redundant or overlapping DL-ACUs, preventing distortion in the NLI scoring and aggregation phases.

5.2. Selective Document-Atomic Content Unit

The Selective Document-Atomic Content Unit (SD-ACU) module also shows notable performance gains, though the impact was not uniform across all datasets. We observe a more pronounced trend of improvement on benchmarks rich in extractive summaries. For instance, the gains are more evident on the CNN/DM portion of the AggreFact benchmark (Table 3), the MultiNews and GovReport subsets within DiverSumm (Table 4), and most constituent datasets of the SummaC benchmark, with the exception of the highly abstractive XSumFaith (Table 1).

The quantitative results further substantiate this trend. On the largely extractive SummaC benchmark (Table 1), the ‘FENICE + SD-ACU’ configuration achieves the highest average balanced accuracy of 77.12%, which is 1.36 percentage points higher than the baseline FENICE. This strong performance highlights the module’s effectiveness in contexts where direct, fine-grained alignment between summary and document facts is most beneficial.

The effectiveness of this module is illustrated with an example from the MultiNews dataset, where a summary sentence appears verbatim in the source document: “Action News Jax spoke to Professor Tiri Fellows, who is deaf and teaches American Sign Language.” Baseline decomposers like ORCA-2_FIZZ and T5-base_FENICE extract multiple ACUs from this summary, including ACU “Tiri Fellows is deaf.” In an NLI scoring phase in Figure 1, this simple ACU (as the hypothesis) is compared against the entire, complex document sentence as the premise, which contains other potentially distracting facts such as “Action News Jax spoke to Professor Tiri Fellows” or “Professor Tiri Fellows teaches American Sign Language.”. In contrast, our SD-ACU method decomposes the document sentence into its own atomic content units. This enables a far more precise ACU-to-ACU alignment by creating a perfectly matched premise. In this case, the premise also becomes “Tiri Fellows is deaf.”. This symmetrical alignment provides a much cleaner and more robust signal to the NLI model. We hypothesize that this is because symmetrically matching the granularity of both the document and the summary is particularly beneficial when summary facts have direct counterparts in the source text, simplifying the factuality verification task.

5.3. Synergistic Effects and Overall Performance

When both modules are combined, they often exhibit a synergistic effect, addressing a wider range of factual inconsistencies. This is particularly evident on the FaithBench benchmark (Table 5), where the full ‘DL-ACU + SD-ACU’ model achieves the highest ROC-AUC for both the FIZZ and FENICE families, demonstrating consistent, incremental improvements. Similarly, on the AggreFact-FtSota benchmark (Table 3), the combined ‘FENICE + DL-ACU + SD-ACU’ model obtains the best overall average balanced accuracy among all systems. These results suggest that the two modules are complementary. The DL-ACU captures complex discourse link errors, while the SD-ACU improves the precision of atomic fact verification, especially in extractive contexts.

6. Conclusions

Recent research on hallucination detection in summarization involves decomposing the summary into atomic content units (ACUs) and then verifying the factuality of each ACU in relation to the original text. However, as the ACUs generated in this process become increasingly shorter, limitations arise in that discourse link information such as temporal order, causality, and purpose are not taken into consideration. To address this limitation, this study converts the summary into an AMR graph and extracts Discourse Link-Aware Content Units by constructing subgraphs centered on discourse link relations. In addition, unlike previous studies that decomposed only the summary into ACUs, this study selectively decomposes only specific parts of the document corresponding to the summary ACUs, enabling precise entailment evaluation at the ACU-to-ACU level. As a result, the accuracy of NLI-based hallucination detection in summarization systems is improved.

The experiments are conducted using the AggreFact-FtSota, DiverSumm, FaithBench, and SummaC benchmarks, showing performance improvements over the FIZZ and FENICE baselines in most cases. In particular, for the discourse link error type LinkE in the FRANK, a significant reduction in the error rate is achieved compared to other error types. This study is inspired by the requirement that an ACU must not deviate from the summary’s content while fully incorporating all conveyed meaning. We propose a Discourse Link-Aware Content Unit based on the AMR graph, complemented by our Selective Document-Atomic Content Unit approach for precise and efficient alignment, and demonstrate that these methods together can address the fundamental limitations of recent NLI-based detection methods that decompose the summary into factually incomplete ACUs.

7. Limitations

Our approach yields consistent gains on benchmarks where discourse link errors are explicitly present, but the improvements are less stable on benchmarks that do not intentionally inject such errors. This suggests that the method is relatively specialized for discourse link error. The contribution is a simple augmentation that adds atomic content units carrying discourse links to the pipeline, and it primarily targets the LinkE category in FRANK, which limits coverage of other types of factual errors.

Given that recent ACU decomposition typically relies on generative models, it is important to ensure both conciseness and completeness between the summary and its ACUs. Here, conciseness means that each summary ACU must be supported by the summary itself without introducing new information or over-extending the interpretation. We enforce this criterion by applying an NLI-based filter that checks whether each ACU is entailed by the summary. In contrast, completeness means that the set of ACUs should collectively capture all of the meaning conveyed by the summary. One frequent omission is the discourse link relation addressed in this work, but other information can also be missed. In particular, modality and hedging (e.g., uncertainty markers), attribution and perspective, and non-core semantic roles (e.g., manner, condition, concession) may still be lost by decomposition, while our module restores discourse links, an ACU-free approach could further mitigate such losses. A more systematic investigation of such omissions, together with principled methods to assess and mitigate them, remains necessary.

Finally, because a modular pipeline (AMR parsing → coreference resolution → NLI filtering) can accumulate mismatches and propagate errors across components, an end-to-end verifier that jointly parses, aligns, and judges factuality is a compelling future direction. Such a model could reduce threshold sensitivity and component-level brittleness.