KoDocXAI: A Quantitative Evaluation Framework and Benchmark for Explainability in Key Information Extraction from Korean Administrative Form Documents

Featured Application

The proposed framework can be used in document processing pipelines for public administration and finance to verify the evidence behind automatically extracted values and to triage cases that require human review.

Abstract

Key information extraction (KIE) models are increasingly used in high-stakes document workflows, where each extracted value must be accompanied by evidence a reviewer can verify. Evaluating their explanations is difficult: human rationale annotation is costly, and in agglutinative Korean, subword tokens misalign with the units humans read. We present KoDocXAI, an evaluation framework and pilot benchmark for the explanations of KIE models on Korean administrative form documents. It derives a value-region rationale reference from existing KIE bounding-box annotations and scores value-region plausibility, whether or not an explanation localizes the extracted value; re-aggregates subword attributions to morphemes and eojeols (Korean spacing units); and measures faithfulness by modality-wise perturbation with the pretraining [MASK] token while protecting the ground-truth span. The pilot benchmark v1.0 covers five fields over 300 permit-and-authorization dispatch documents, 20% of which are doubly annotated. With a fine-tuned LayoutXLM, Integrated Gradients exceeds a random baseline on both axes, whereas attention rollout stays at baseline level under both a first-token and a span-level target. Span protection removes inflated faithfulness scores and a spurious aggregation-level effect. The modality interventions differed in strength, but under them the image channel contributed practically nothing, confining the validated scope to text and layout. These results hold for this document type and model; they support validating the measurement protocol itself and reporting explainability separately by axis and modality.

1. Introduction

Document understanding is the task of analyzing not only the text contained in a document, but also its layout, visual elements, and their interrelations, in order to interpret the document’s meaning and structure. Recent advances in pretrained models that integrate textual and visual information, and in end-to-end models that dispense with optical character recognition (OCR), have substantially improved performance across document understanding tasks such as form understanding, receipt information extraction, and document question answering [1,2,3].

High predictive performance, however, does not imply that a model’s decision basis is sound. In settings such as public administration, finance, logistics, and contracting—where extracted information feeds directly into subsequent decisions and business processes—it must be possible to verify not only whether the model extracted a value correctly, but also which text, positions, and visual structures in the document the value was grounded in. When the same amount or date appears several times in a document, for instance, ordinary accuracy metrics cannot distinguish whether the model produced its output from the correct field–value relation or merely happened to select an identical string. High-stakes document environments therefore require explainability evaluation that can verify the model’s decision basis alongside prediction accuracy.

These requirements are reinforced by changes in the institutional environment, not only by concerns internal to the technology. As AI adoption expands in the public sector, the accountability question of whether values extracted by AI from administrative documents may serve as the basis of administrative acts is being raised. Korea’s Framework Act on the Development of Artificial Intelligence and the Creation of a Foundation for Trust (Act No. 20676; the “AI Framework Act”), effective January 2026, imposes an advance-notice obligation on services based on high-impact or generative AI (Article 31), and requires operators of high-impact AI to establish measures for explaining, to a technically feasible extent, the final results and main criteria used to derive them (Article 34); this Act’s definition of high-impact AI includes systems used in decision-making by state agencies [4]. Likewise, the EU Artificial Intelligence Act stipulates transparency requirements for high-risk AI systems (Article 13) and a right to explanation for persons affected by decisions based on them (Article 86) [5]. In parallel, as the digitization of administrative records subject to long-term preservation proceeds, so does the demand to provide information extracted from archived documents in a verifiable form. A tool for evaluating the explainability of AI over public documents addresses both demands.

Three obstacles stand in the way of such evaluation. Standard plausibility evaluation compares explanations with human-annotated rationales, which are costly to produce and absent from existing Korean document resources; standard faithfulness evaluation was developed for single-modality text or image inputs and does not separate the contributions of text, layout, and image; and token-level evaluation presupposes that tokens are units a reader can interpret, which does not hold for agglutinative Korean. What is needed, and what this paper provides, is an evaluation protocol that works from the annotations that KIE datasets already contain, keeps the modalities apart, and scores explanations in units that a Korean reader can verify.

To address these problems, we propose KoDocXAI, a framework for evaluating the explanations that attribution methods produce for KIE models on Korean administrative form documents; it evaluates explanations and does not itself propose an extraction method. Its quantitative layer is validated empirically in this paper, and its qualitative layer is specified as a proposed protocol. We take the region of the ground-truth value in the KIE annotations as a value-region rationale reference and evaluate how well attributions localize that region (value-region plausibility); whether the model relies on other evidence, such as field labels, is measured separately on the faithfulness axis. We further apply text masking, layout coordinate perturbation, and image region occlusion to the selected evidence, measuring the effect of the text, layout, and image modalities on the actual prediction. To reflect the linguistic properties of Korean, subword-level attributions are re-aggregated to the morpheme and eojeol (the Korean word-spacing unit) levels, and for semantic consistency and user verifiability—dimensions that resist automation—the framework specifies a scoring rubric and an application protocol. A pilot application with three non-author raters is reported as a feasibility check, and validation of the layer is left to future work. KoDocXAI (Korean Document XAI) is an open evaluation framework and resource suite for explainability in Korean document understanding, instantiated here for administrative form documents. Its parts are referred to as the KoDocXAI framework, the KoDocXAI benchmark (v1.0), and the KoDocXAI toolkit.

This study poses three research questions.

RQ1. Does the aggregation level of attribution (subword, morpheme, eojeol) change the outcome of explanation evaluation, and under what conditions can such an effect be observed?

RQ2. Can the value-region rationale reference derived from KIE annotations serve, without separate human rationale annotation, as a reference for quantifying the value-region plausibility of explanations?

RQ3. Can prediction sensitivity to the text, layout, and image modalities be measured separately? Does the modality sensitivity pattern differ across fields?

RQ1 is answered in Section 4.2, RQ2 in Section 4.3, and RQ3 in Section 4.4.

The main contributions of this study are as follows.

• A framework for evaluating the explanations of KIE models with multimodal input (text, layout, image) using modality-wise diagnostics: value-region plausibility scored against a value-region rationale reference derived from existing KIE bounding-box annotations; faithfulness measured by modality-wise perturbation with the ground-truth span protected; and re-aggregation of subword attributions to Korean morphemes and eojeols, which also serves as a control that isolates artifacts of the evaluation protocol.
• The KoDocXAI benchmark v1.0, a pilot release of 300 Korean permit-and-authorization dispatch documents annotated with field type, value span, and bounding boxes for five fields (20% doubly annotated, with inter-annotator agreement reported); released together with the annotation guideline and annotation tool.
• Empirical findings on this benchmark with a fine-tuned LayoutXLM: the reference discriminates attribution methods; protecting the ground-truth span removes inflated faithfulness scores and a spurious aggregation-level effect (self-evidence masking); and the image channel is inert in this setting, a property that only modality-wise reporting exposes.
• A qualitative-layer rubric and application protocol for semantic consistency and user verifiability, presented as a proposed design, with a pilot application with three non-author raters reported as a feasibility check; validation of the layer is left to follow-up work.

Section 2 reviews related work. Section 3 describes the evaluation framework and the construction of the KoDocXAI benchmark. Section 4 reports the experimental results, Section 5 discusses their implications and threats to validity, and Section 6 concludes with directions for extension.

2. Related Work

2.1. Document Understanding Models and Key Information Extraction

Document understanding models have advanced through pretraining that combines text, layout, and image. The LayoutLM family established the approach of coupling token representations with two-dimensional coordinates [1]: LayoutLMv2, which integrates a visual backbone, and LayoutXLM, its extension to 53 languages, consume the three modalities jointly [6,7]. LayoutLMv3 learns joint linguistic and visual representations through unified text–image masking and word–patch alignment [2], and Donut removes the OCR stage and generates structured outputs directly from document images [3]. Key information extraction is a representative downstream task of these models and has been evaluated mainly on public benchmarks such as FUNSD, SROIE, XFUND [8,9,10], and CORD [11]. For Korean documents, the public KIE benchmark identified by the authors is KORIE, which targets the receipt domain [12]. For the present study, these models matter in two ways: LayoutXLM is the family member pre-trained on Korean that consumes text, layout, and image jointly, which a modality-wise evaluation requires; and none of the listed benchmarks carries rationale annotations, so an evaluation of explanations must obtain its reference from the KIE annotations themselves.

2.2. Explanation Attribution and Faithfulness Evaluation

In explainable artificial intelligence (XAI), evaluation conventionally distinguishes faithfulness (how truthfully an explanation reflects the model’s actual decision process) from plausibility (how well the presented explanation matches the rationale a human would consider reasonable). The ERASER benchmark proposed metrics that compare model explanations against human-annotated rationales, together with comprehensiveness and sufficiency, which measure how much the prediction changes when the selected rationale is removed or retained [13]. Findings that attention weights are difficult to regard as explanations in themselves further show that the visual or intuitive persuasiveness of an explanation is not enough; whether the cited evidence actually influenced the prediction must be separately verified [14]. Two design decisions follow. Because human rationales of the ERASER kind are unavailable for Korean documents and costly to create, we derive the reference from the value annotations and name the resulting metric for what it measures (value-region plausibility, Section 3.3.1); and because attention weights cannot be taken as explanations in themselves, attention-based attribution is compared against a random baseline on both axes rather than reported alone.

Representative attribution methods include the gradient-based Integrated Gradients (IG) [15] and the attention-based attention rollout [16]. In perturbation-based faithfulness measurement, physically deleting tokens is known to push the input out of the training distribution, so that what is measured is the response to anomalous input rather than evidence importance; FUD, which guarantees in-distribution perturbation via diffusion-based reconstruction [17], and Normalized AOPC, which corrects scale bias across attribution methods [18], were recently proposed. The quantitative layer of this study extends this lineage of metrics to the multimodal document setting and to the unit system of an agglutinative language. We adopt the pretraining [MASK] token as the replacement value because reconstruction-based perturbation is too costly for repeated document-level evaluation, and we verify empirically that the method ranking is unchanged under token deletion and random substitution (Section 4.2); NAOPC is not required because every comparison between attribution methods is made within one model at a time (the second backbone described in Section 4.3 is compared only on the sign of its within-model contrasts).

2.3. Explainability in the Document Domain

Explainability research is also emerging in the document domain. Recent work has compared several document image classification models and explanation methods, reported substantial differences in the faithfulness and interpretability of existing techniques, and proposed a document-specific explanation method that directly removes document components to measure importance [19]. These studies, however, mainly target document image classification. In document understanding tasks such as KIE, the prediction target is a specific field and value rather than a whole-document class, and the model simultaneously exploits text, two-dimensional position, and image features. Existing XAI evaluations designed for a single image region or a single text token do not transfer directly, and evaluation methods are needed that can separately verify how the different modalities contribute to a prediction. The design consequence is that evaluation must be defined at the level of a field–value prediction and must perturb each modality separately, which is the structure described in Section 3.3.

Meanwhile, research on explanation generation and evidence citation is growing rapidly in the document visual question answering (DocVQA) line [20,21]. CiteVQA, which benchmarks the reliability of a model’s self-cited evidence, demonstrated that the appearance and substance of evidence citation can differ [22]. This study differs from that line in three respects. First, its target includes the KIE task and encoder-based models rather than DocVQA and generative vision–language models. Second, what is evaluated is not a model’s self-citation ability but the quality of the explanations produced by attribution methods, independent of any particular model architecture. Third, to the best of our knowledge, no prior work has systematically evaluated explanation faithfulness in document understanding models using modality-wise perturbations separately applied to text, layout, and image information.

2.4. Korean Resources and Linguistic Properties

In Korean documents, these problems become more intricate. Korean is an agglutinative language in which josa (postpositional particles) and verbal endings attach to stems, and the subword token boundaries used by neural language models need not coincide with the morpheme or eojeol boundaries that humans interpret. KLUE likewise reported that combining morpheme-level pre-tokenization with BPE is effective for morpheme-level Korean tasks [23]. If subword tokens are used as-is as semantic units in explainability evaluation, a single meaningful piece of evidence may be scattered across several tokens, and particles or endings may receive independent importance scores; therefore, a definition of explanation units appropriate for Korean documents is required. In form documents, these linguistic units correspond to positions and visual regions on the page, so textual units and spatial evidence must be considered together.

There is also a resource gap. Among the resources identified in our review, the only Korean KIE benchmark is KORIE, which targets the receipt domain [12]; we found no Korean evaluation resource that covers form documents in general while supporting explainability evaluation with box-level rationale annotations. Table 1 summarizes the prior approaches by model, explanation type, evidence level, perturbation type, and availability of human rationales, and positions the present framework among them.

Table 1. Prior work related to this study, categorized by the models or tasks targeted, type of explanation, level at which evidence is defined, perturbation used for faithfulness, and whether human-annotated rationales are used. n/a: not part of that work.

Work	Target Model/Task	Explanation Type	Evidence Level	Perturbation for Faithfulness	Human Rationales
ERASER [13]	Text classification and QA (any model)	Extracted text rationales	Token or sentence	Removal/retention of the rationale (comprehensiveness, sufficiency)	Yes (annotated)
Jain and Wallace [14]	Attention-based text models	Attention weights	Token	Counterfactual attention distributions	No
Integrated Gradients [15]	Any differentiable model	Gradient-based attribution	Input feature	n/a (axiomatic method)	No
Attention Rollout [16]	Transformer encoders	Accumulated attention flow	Token	n/a	No
FUD [17]	Attribution evaluation (image, text)	Evaluation of attributions	Input feature	Diffusion-based in-distribution reconstruction	No
Normalized AOPC [18]	Attribution evaluation (text)	Evaluation of attributions	Token	Removal with scale normalization	No
DocXplain [19]	Document image classification	Model-agnostic component importance	Document component (image region)	Component removal	No
DocVQA Explanation line [20,21]	Document VQA (vision–language models)	Model-generated visual or textual explanation	Image region or text	n/a	No
CiteVQA [22]	Document VQA (vision–language models)	Model self-cited evidence	Cited region or text	n/a	Yes (annotated evidence)
KORIE [12]	Korean receipt KIE (benchmark)	n/a	n/a	n/a	No
KoDocXAI (this study)	Korean form-document KIE (encoder model, LayoutXLM)	Post-hoc attribution (IG, rollout; Gradient × Input and occlusion as checks) re-aggregated to morpheme and eojeol	Value-region reference derived from KIE boxes	Modality-wise: [MASK] text replacement with span protection, layout jitter and swap, image occlusion	No (value-region reference; human validation left to future work)

3. Materials and Methods

3.1. Problem Formulation

Key information extraction over form documents takes as input a document image D, a token sequence T = (t₁,…,t_n) obtained by OCR, and the corresponding bounding boxes B = (b₁,…,b_n), and identifies in the document the value span corresponding to each field in a predefined field-type set $ℱ$ = {f₁,…,f_K}. Our experiments target encoder-style document understanding models formalized as token-level BIO tagging, while the evaluation framework itself is designed to be independent of any particular model architecture.

Attribution is defined as a function that assigns a contribution score to each input unit for a specific prediction of a model f. Given the predicted value span ${\widehat{y}}_k$ for field f_k, an attribution method A produces a score vector $A(f,x,{\widehat{y}}_k)=({\varphi }_1,\dots ,{\varphi }_n)$, ϕ_i ∈ ℝ. In evaluation, this prediction target is operationalized as the label sequence the model actually predicted at the positions of the ground-truth value span (Section 3.5), so an explanation target exists even for fields where the model predicts no span (the predicted span notation below denotes this operational target). The input units are primarily the model’s own processing units, subword tokens; Section 3.2 defines the procedure that re-aggregates them to the morpheme and eojeol levels. The object of evaluation is not an individual model but the pair (model f, attribution method A). This distinction separates our framework, at the level of what is evaluated, from the recent line of work that benchmarks a model’s own ability to cite evidence [22].

The framework starts from deriving a value-region rationale reference directly from KIE annotations, without separate human rationale annotation. The region where the value of field f_k actually lies on the document—the union of the bounding boxes of the tokens constituting the ground-truth value span, $R_k={\bigcup }_{t_i\in y_k}{b}_i$—serves as the reference region. We take the value region as the reference because, in a verification setting, the role of an explanation is to show a reader where the extracted value is located; this axis therefore measures whether an explanation localizes the value, and we name it value-region plausibility. Because it is derived directly from the KIE annotation structure, it requires no separate human rationale annotation (RQ2) and removes the additional cost of rationale-specific annotation (the cost of the KIE annotation itself remains). The reference does not cover other legitimate evidence a reader might use, such as field labels or surrounding context. Whether the model relies on such evidence is measured along the faithfulness axis, by context perturbation (Section 3.3.2) and the context sensitivity analysis described in Section 4.4. The sensitivity of the value-region plausibility results, to the extent defined by the reference region, is examined in Section 5.2.

3.2. Redefining Attribution Units for Korean

Many text-XAI evaluations operate at the model-token level or aggregate attributions to whitespace-delimited words. In Korean, an agglutinative language, neither unit maps cleanly onto a unique linguistic unit, for two reasons. First, a single eojeol combines content morphemes (nominal and verbal stems) with functional morphemes (particles and endings), so eojeol-level attribution lumps distinct linguistic functions into one score. Second, subword tokenizers segment text without regard to morpheme boundaries, so subword-level attribution assigns scores to units that are not linguistically meaningful.

We measured this on the target corpus. Segmenting the 36,655 eojeols of the KoDocXAI benchmark v1.0 with LayoutXLM’s multilingual SentencePiece tokenizer yields 96,058 subwords, of which 6.3% cross a morpheme boundary; eojeols containing at least one such subword account for 15.0% of the total. In the opposite direction, 29.8% of the 74,874 morphemes are split across two or more subwords. For example, the lot number eojeol “740-5번지가” (740-5-beonji-ga) is analyzed into the morphemes 740/SN, -/SY, 5/SN, 번지/NNG (“lot number”), and 가/JKS (nominative particle), whereas the tokenizer splits it into “▁”, “740”, “-5”, “번”, and “지가”. The final token “지가” fuses the latter half of the noun 번지 with the nominative particle 가, so whether that token’s attribution score stems from the meaning of the value (“lot number”) or from grammatical function (nominative marking) is unidentifiable by definition. Conversely, in the eojeol “등록하였음을” (deungnok-hayeoss-eum-eul, “that [it] has been registered”), the token “하였” straddles the derivational suffix 하/XSV and the pre-final ending 였/EP.

We therefore define a procedure that re-aggregates the basic unit of attribution from subwords to morphemes and eojeols; perturbation is then applied to the eojeols that the selected units belong to (Section 3.3.2). A morphological analyzer (MeCab-ko [24]; python-mecab-ko 1.3.7 with mecab-ko-dic 2.1.1) decomposes each eojeol w into a morpheme sequence (m₁,…,m_J), and character-offset alignment computes the overlap between subword τ and morpheme m. The morpheme-level attribution score is the overlap-weighted sum:

$$\varphi (m)={\sum }_{\tau }{\displaystyle \frac{\left|\mathrm{overlap}(\tau ,m)\right|}{\left|\tau \right|}}\varphi (\tau ),$$

(1)

and the eojeol-level score is $\varphi \left(w\right)={\sum }_{m\in w}\varphi \left(m\right)$. The same attribution vector can thus be compared at three aggregation levels—subword, morpheme, and eojeol—and the effect of the aggregation level on evaluation outcomes becomes the manipulated variable of this study (RQ1).

Part-of-speech tags further divide morphemes into functional (particles, J*; endings, E*) and content (nominal and verbal stems, among others) categories. Running the faithfulness measurements under three masking conditions—(i) all morphemes, (ii) content morphemes only, and (iii) functional morphemes only, each restricting the morphemes eligible for ranking—quantifies how far the evidence reached through functional morphemes affects faithfulness metrics.

Operational definition of morpheme categories. mecab-ko-dic occasionally outputs an inflected form as a single morpheme, producing composite tags that combine a stem and an ending (바랍니다/VV + EF, 된/XSV + ETM, etc.). Such units cannot be split further at the surface level: in “바랍니다” the ending “ㅂ니다” lies inside a syllable block, so no character offset can draw the boundary. We classify a morpheme as functional-only when every component of its tag is a particle or an ending, and as content-only when no component is; mixed morphemes combining a stem and an ending are excluded from both conditions (ii) and (iii). This keeps the two conditions mutually exclusive and pure, at the cost that (ii) and (iii) do not sum to (i). In the KoDocXAI benchmark v1.0, the three categories account for 88.1% (content), 9.0% (functional), and 2.9% (mixed).

Scope restriction from corpus properties. A functional-morpheme ratio of 9.0% is low for Korean text in general. The target documents are administrative forms composed of titles, recipients, document numbers, and tabular items, so noun phrases dominate and full sentences are rare. The aggregation-level effect observed in this study is therefore close to a lower bound under a low functional-morpheme ratio, and the effect is expected to be larger in document types with a higher share of running text. We revisit this point together with generalizability in Section 5.

3.3. Two-Layer Evaluation Framework

The proposed framework consists of a quantitative layer of automatically computable metrics and a qualitative layer for dimensions that cannot be automated (Figure 1). The quantitative layer covers three metric families—value-region plausibility, faithfulness, and modality-wise perturbation sensitivity—while the qualitative layer covers two dimensions: semantic consistency and user verifiability. The quantitative layer enables large-scale, repeatable evaluation, and the qualitative layer addresses trust judgments in real usage contexts. The contribution validated in this paper is the quantitative layer (RQ1–RQ3, Section 4); the qualitative layer is presented as a proposed rubric and application protocol (Section 3.3.4), with a pilot application reported in Section 5.5 and Supplementary Document S2, and its validation left to follow-up work.

Figure 1. The two-layer explainability evaluation framework. The value-region rationale reference R_k is derived directly from key information extraction (KIE) annotations (RQ2), and attributions are re-aggregated from subwords to morphemes and eojeols (RQ1). The three metric families of the quantitative layer are validated in this paper; for the qualitative layer, the rubric and application protocol are presented as a proposed design and have been applied only in a pilot study (Section 3.3.4 and Section 5.5). Arrows indicate the flow from the inputs to the metrics; the colored boxes and arrows mark the components addressed by RQ1 (green), RQ2 (orange), and RQ3 (blue), and the dashed box marks the qualitative layer.

3.3.1. Value-Region Plausibility

We measure the spatial agreement between the bounding-box region indicated by the top-k attribution units and the value-region rationale reference R_k. Because the unit-to-box correspondence is fixed deterministically via the OCR boxes, value-region plausibility is computed as intersection over union at k (IoU@k) and set-based F1. As an auxiliary metric, we use pointing accuracy (Hit@1), which checks whether the single top attribution unit lies inside the reference region. Value-region plausibility measures whether an explanation localizes the value that a reviewer must verify; it is independent of whether the model actually relied on those locations (Section 3.3.5), and it does not score evidence outside the value region (Section 3.1).

3.3.2. Faithfulness

Faithfulness is measured by perturbation-based comprehensiveness and sufficiency. Writing the prediction probability as $p_f({\widehat{y}}_k\mid x)$,

$$\mathrm{COMP}@k=p_f({\widehat{y}}_k\mid x)-p_f({\widehat{y}}_k\mid x_{\backslash \text{top-}k}),$$

(2)

$$\mathrm{SUFF}@k=p_f({\widehat{y}}_k\mid x)-p_f({\widehat{y}}_k\mid x_{\text{top-}k}).$$

(3)

A known pitfall is that physically deleting tokens pushes the input outside the training distribution (out-of-distribution, OOD), so that the measurement reflects the response to anomalous input rather than evidence importance. We adopt low-cost pretraining-consistent masking: instead of being deleted, tokens are replaced with the [MASK] token observed during pretraining. In faithfulness perturbation, the eojeols of the ground-truth value span form a protected set that is never masked under any condition; the top-k units to be masked are selected among non-protected units only, so protected eojeols never enter the candidate ranking. Masking is applied at eojeol granularity at every aggregation level: the ranked units (subwords, morphemes, or eojeols) are mapped to the eojeols they belong to, duplicates are skipped, and the next-ranked units fill the k slots, and every subword token of each selected eojeol is replaced by [MASK]; no subword is ever partially masked, so the aggregation level and the masking condition change which eojeols are selected, not the granularity of masking. When fewer than k eligible eojeols exist, which occurs only under the functional-only condition (8.8% of instances at k = 10), the available ones are masked. Erasing the value text itself would trivially collapse the prediction and render the metric degenerate, so our faithfulness measures the model’s reliance on contextual evidence outside the value span. Accordingly, SUFF keeps the protected set together with the top-k units; this definition is a task-adapted variant of standard comprehensiveness and sufficiency.

The recently proposed FUD [17] guarantees in-distribution perturbation via diffusion-based reconstruction, but it is computationally prohibitive for repeated document-level evaluation. The case that replacement value masking stays closer to the pretraining distribution than deletion or random substitution rests, by design, on two grounds. First, the replacement token [MASK] is an input the model observed repeatedly under the masked-language-modeling objective of pretraining, so it is an input the model was trained to process; this does not by itself guarantee that an input in which several units are masked at once is in-distribution as a whole. Second, our perturbation replaces only the text stream while each eojeol’s bounding box and the document image keep their original values, so the input distributions of the layout and visual modalities are unchanged by the perturbation. These two grounds are design-level arguments. As an empirical check, Section 4.2 presents a repeat of the faithfulness evaluation with two alternative text perturbations, token deletion and random in-vocabulary substitution, on a fixed subset of documents; a comparison with reconstruction-based perturbation remains future work.

Curves obtained by varying k are summarized by AOPC, computed here as the unweighted mean of COMP@k over the predefined grid k ∈ {1, 3, 5, 10}, a discrete variant of the area over the perturbation curve, not an integrated area. We do not compute Normalized AOPC [18], which corrects scale bias mainly to support cross-model comparison: the method comparisons in this study are made within one fine-tuned model at a time, at the same aggregation level and masking condition, against the random baseline, so we report AOPC values—changes in prediction probability—and defer NAOPC to the multi-model extension discussed in Section 5.5.

3.3.3. Modality-Wise Perturbation Sensitivity

Document understanding models consume text, layout, and image jointly, so faithfulness is likewise decomposed by modality. The text modality is perturbed by unit-wise masking as in Section 3.3.2; the layout modality by jittering the coordinates of the reference region (±δ) and by swapping coordinates between fields; and the image modality by occluding patches of the reference region. The principle of perturbation is to remove each modality’s information channel rather than the identity of the evidence. Text masking erases the identity of the value itself, so the ground-truth value span is excluded as a protected set (Section 3.3.2), whereas layout and image perturbations preserve the textual identity and therefore target the ground-truth region directly. Perturbing the top-k attributed region diagnoses the faithfulness of an attribution method, while perturbing the ground-truth region R_k diagnoses the model’s modality dependence; which criterion each metric uses is stated alongside the results. The jitter magnitude ±δ is scaled to the observed natural variation: three levels are used, set to approximately 0.4×, 1×, and 2×, reflecting the standard deviation of the natural coordinate variation measured on the corpus (exact values in Section 4.4). Aggregating the three sensitivities into a single index entails an arbitrary choice of weights, which we place outside the scope of this paper; modality-wise reporting is the principle. The three interventions remove different amounts of information, so their sensitivities are not comparable in strength across modalities and are read as separate diagnostics (Section 4.4).

3.3.4. Qualitative Layer

Two dimensions that resist quantification are handled by rubric-based expert evaluation. Semantic consistency scores the degree to which an explanation supports the correct contextual distinction when values of the same type must be assigned to different fields (e.g., several dates in one document serving respectively as the issuance date and as an in-body deadline); user verifiability scores how easily a reviewer can detect and reject model errors when explanations are presented. Both use a five-point scale. This paper presents the rubric definitions and design rationale together with the application protocol (number and qualifications of raters, training procedure, and the computation of inter-rater agreement); a pilot application with three non-author raters is reported in Section 5.5 and Supplementary Document S2, and validation with the rater groups that the protocol specifies is left to follow-up work. Because the objects scored by the qualitative layer are explanations produced by a model, the pilot study used the outputs of the quantitative-layer experiments as its input. The full scoring rubric and protocol are provided in the Supplementary Materials so that third parties can reproduce them identically.

3.3.5. Reporting Principle

Value-region plausibility and faithfulness are independent axes with no guaranteed correlation. An explanation may plausibly point to the ground-truth region while being unrelated to the model’s actual dependence structure (high value-region plausibility, low faithfulness), and the converse also holds. We therefore report the two axes separately rather than collapsing them into a single score, and analyze cases of divergence between them in Section 4 (Results). This separate-reporting principle shares its concern with recent evidence that the appearance and the substance of evidence citation can differ [22].

3.4. Constructing the KoDocXAI Benchmark

The source data are taken from the AI-Hub Public Administrative Documents OCR dataset (공공행정문서 OCR; constructed in 2020; www.aihub.or.kr). On the criteria of a form-like field–value structure, controllability of personal information, and layout regularity, we selected a single document type: permit-and-authorization official dispatch documents (sihaengmun, the officially dispatched copies of approved drafts). The original data provides transcribed text and bounding-box annotations but no field-type annotation. We therefore defined a five-field schema (title, recipient, document number, issuance date, issuing authority, with the last field denoting the head-of-agency signature at the bottom of the page, not the agency letterhead; Section 3.4.1) and annotated each document with 1:1 links of field type, value span, and bounding boxes as the value-region rationale reference.

Although the sample was produced between 1994 and 2002, the five fields are those that government administrative work regulations have prescribed for dispatch documents since the Office Management Regulation (Presidential Decree, enacted 1991), in force during the sample period, and still prescribe under its successor, the current Regulations on the Operation and Innovation of Administrative Work (renamed in 2023) [25]. In current electronic dispatch documents, the recipient and title remain in the header, and the issuing authority remains the head-of-agency signature at the bottom, whereas the document number and issuance date are now printed in the dispatch line at the foot of the page, and the approval box of the sampled era has been replaced by electronic approval. Within the sampled corpus, the header layout is highly regular: in a pre-sampling survey of 4228 documents, 92.2% showed an identical header layout. The benchmark therefore represents a standard field schema that persists in current practice, realized in the layout of the sampled era, while retaining the realistic conditions of archived records, including official seals and low-quality scans; whether the numerical findings replicate in contemporary electronic documents is addressed in Section 5.5.

The “reference (참조)” field was excluded from the schema: it appears in 80.3% of documents but actually carries a value in only 26.1%, which would make it a field with an excessive absence rate.

The KoDocXAI benchmark v1.0 is a pilot release of 300 documents with 1173 field instances carrying a present value (1500 slots, including 327 absences). The sample was drawn by proportional allocation from 18 strata crossing 3-year bins with title series, and the presentation order was randomized with a fixed seed so that annotator practice effects would not concentrate in particular strata. Annotation was performed with a web-based annotation tool built for this study, and 20% of the documents (60) were doubly annotated. Inter-annotator agreement (IAA) reached a Cohen’s κ of 1.000 for field presence in every field, and the eojeol-set Jaccard value span was 0.968–1.000 (Table 2). The v1 release is scoped to framework validation; the roadmap for extending document types and scale is discussed in Section 6. The annotation schema and an example are shown in Figure 2, and the descriptive statistics are shown in Table 2.

Table 2. Per-field descriptive statistics and inter-annotator agreement of the KoDocXAI benchmark v1.0. The absence rate is the proportion of the 300 documents in which the field does not exist. “String-irrecoverable” denotes cases in which the region is identified but the string cannot be recovered (Section 3.4.1); such cases remain in the value-region plausibility sample and are excluded only from string match scoring. IAA was computed on the 60 doubly annotated documents.

Field	Present	Absent	String-Irrec.	Absence Rate	Mean Length (Chars)	Mean Eojeols	κ	Status Agr.	Exact Span	Span Jaccard	String Match
Title	282	18	6	6.0%	13.1	3.5	1.000	96.7%	100.0%	1.000	100.0%
Recipient	295	5	17	1.7%	12.8	3.1	1.000	98.3%	98.3%	0.997	98.2%
Document number	197	103	21	34.3%	11.4	2.6	1.000	98.3%	97.2%	0.986	100.0%
Issuance date	197	103	43	34.3%	7.6	2.8	1.000	98.3%	91.7%	0.968	100.0%
Issuing authority	202	98	45	32.7%	4.1	3.5	1.000	96.7%	100.0%	1.000	100.0%

Figure 2. Example of the KoDocXAI annotation schema (document 5350109-1997-0001-0034). Colors distinguish field types; solid lines are eojeol-level bounding boxes b_i, whose union forms the value-region rationale reference R_k. Dashed lines show the enclosing rectangle for reference, which grows beyond the union when a value wraps across lines, or its eojeols lie apart. The issuing authority “김 해 시 장” (Gim hae si jang, eojeol-by-eojeol transliteration; Mayor of Gimhae) exemplifies a single word split into four eojeols, typical of Korean optical character recognition (OCR) labels, and the circular official seal at the bottom marks the position where the systematic loss described in Section 3.4.1 occurs. Personally identifiable information (staff names, phone and fax numbers, the applicant company’s name and lot number address, and handwritten signatures) is redacted; the five annotated fields and the official seal are left intact.

3.4.1. Annotation Conventions and Scoring Normalization

Given the characteristics of Korean OCR labels, the scoring rules must be stated explicitly, or the benchmark cannot be reproduced. We adopt the following conventions.

Dual storage of value strings. In the original labels, a single word is frequently split across several eojeols (“제 목”, “김 해 시 장”). We store every value in two forms—a canonical form joining eojeols with spaces (value) and a scoring form with spaces removed (value_norm)—and apply the same normalization to predictions at scoring time. Normalization consists of whitespace removal and the stripping of leading and trailing punctuation. The latter absorbs cases where a colon is separated as an independent eojeol, as in “수 신: 김해시…”, observed in 20.1% (title), 21.1% (recipient), 22.6% (document number), and 20.6% (issuance date) of instances. This normalization produced no collisions in which two distinct values merged into one: all 26 groups sharing a normalized value were spacing variants of the same string, such as “김 해 시 장”. Punctuation inside a value is not removed: the hyphen in “공영16000 - 135” and the periods in “1997. 8 . 13” are part of the value, and removing them would fuse the normalized string into forms such as “공영16000135” that mismatch the ground truth. The same rule governs span selection: only punctuation-only eojeols at the edges of a value are excluded, while those in the middle are included.

Distinguishing absence from string irrecoverability. We distinguish fields that do not exist in the document (absent) from cases where the region is clearly identified, but the string cannot be recovered (string_na). The latter remain valid as value-region references R_k, so they stay in the value-region plausibility sample and are excluded only from string match scoring. string_na arises from two causes.

• Seal overlap. Korean official documents bear the official seal stamped over the issuing authority, so characters overlapped by the seal are missed by OCR: in “김 해 시 장”, for instance, the character “김” overlaps an inspection seal and is absent from the word list. This is not random noise but a systematic loss rooted in the formatting conventions of Korean official documents; the text modality is damaged while the layout of the visible (recognized) eojeols remains intact, a naturally occurring analog, for those eojeols, of the modality-wise perturbations of Section 3.3.3.
• Two-line merged labels. The original label occasionally merges characters from two different lines into one eojeol; with no per-character positions inside an eojeol, the unit cannot be split.

Multiple dispatch copies on one page. In documents bearing markers of the form “[제 2 안]” (Draft 2), header fields such as recipient and title appear in several sets. This is not in-body repetition but reflects the practice of producing several dispatch copies for different recipients from a single approval (“one draft, multiple dispatches”). Draft 1 is the internal approval copy, Draft 2 the copy addressed to the applicant, and Draft 3 the copy addressed to related agencies; each draft is a separate dispatch document destined for a different recipient, and the body is abbreviated by reference to the preceding draft (“same as the original”). Of the 300 sampled documents, 52.3% carry such markers, and 23.0% contain two or more header sets on a single page.

Because each draft is a separate dispatch document, concatenating the occurrences into one value yields a string that is the answer of none of them. We take the topmost draft on the page as the ground truth; the spans of lower drafts are preserved under a separate key, not discarded, and are treated in evaluation as ignore, not error: a model pointing to the recipient of Draft 3 is not wrong; it has consulted a different dispatch document. Without this treatment, merging the locations into one span would yield a value string that is the answer of no draft, and a single reference region would span two distinct documents, so which document the model consulted would become undecidable. (Had the reference region been defined as an enclosing rectangle, the merge would have inflated it from 0.5% to 15.3% of the page area, a contrast that motivated the union definition.) Our R_k is the union of eojeol bounding boxes, so no such inflation arises in our metric.

Distinguishing the issuing authority from the letterhead. The agency name (“김 해 시”, Gimhae City) is printed at the top of a dispatch document, while the head-of-agency signature (“김해시장”, Mayor of Gimhae) appears at the bottom. The issuing authority field refers only to the latter; documents without a bottom signature (internally approved drafts, continuation-page-only documents, etc.) are treated as absent. The top letterhead occupies the same position in every document and thus has no discriminative value as a field; admitting it as a value would contaminate value-region plausibility evaluation by turning the letterhead region into ground truth. Before this rule was made explicit, the letterhead was selected as the value in 75 of 300 documents for the first annotator (among the 60 doubly annotated documents, 16 fall in this category, and in 15 of them, the two annotators’ selections coincided); the episode and the resulting changes in agreement are reported in The Effect of Field Definition Ambiguity on Agreement section.

Handling OCR misreads. Misreads in the original labels are left uncorrected: this benchmark evaluates field extraction and its evidence, not OCR accuracy.

The full annotation guideline is released in the Supplementary Materials and at the public repository (https://github.com/kodocxai/kodocxai, accessed on 30 September 2026).

3.4.2. Release

The release (see the Data Availability Statement) follows the terms of use of the source data. AI-Hub data may not be provided or redistributed to third parties without approval, and overseas transfer requires a separate agreement (terms verified on 15 August 2026); the original images and labels are therefore not redistributed. We release only our annotation layer at the public repository (GitHub: https://github.com/kodocxai/kodocxai, accessed on 30 September 2026; Hugging Face: https://huggingface.co/datasets/kodocxai/kodocxai, accessed on 30 September 2026), composed of document identifiers, field types, status labels, annotation-ID index spans referencing the source–label entries, label–file paths, and integrity hashes, so that it contains no original data (images, transcribed strings, or coordinates). Users obtain the originals from AI-Hub and reconstruct the benchmark by joining the annotations with the released script (repository release v1.0). Reassembly is deterministic and manifest-based, and the released scripts include an integrity-verification command. Since applications for the source dataset are limited to residents of the Republic of Korea, full reconstruction is domestically restricted (Section 5.5). In accordance with the attribution obligation for derivative works, the data source is stated in the Data Availability Statement.

The corpus is not released in translation: the evaluation targets properties of the Korean surface form—eojeol segmentation and morpheme–subword alignment—that translation would remove. English documentation is provided throughout: schema and label identifiers, English summaries of the guideline and rubric (Supplementary Documents S1 and S2), and a transliterated, glossed example in Figure 2.

3.5. Experimental Setup

The model under evaluation is LayoutXLM-base [7], a multilingual pre-trained document understanding model, fine-tuned on the training split of permit-and-authorization dispatch documents. LayoutXLM is pre-trained on 53 languages, including Korean, and its LayoutLMv2-style architecture [6] consumes the token, layout, and image modalities jointly, matching our modality-wise perturbation design. The main experiments in this study are conducted on this single model; a second backbone without visual features is used only as a robustness check of the method ranking (Section 4.3). For a cross-lingual contrast, we considered a configuration using a Latin-script receipt resource (CORD [11]) and LayoutLMv3, which is suited to it. In that configuration, however, the model, the pretraining language, the corpus, and the document form change at once, so no difference could be attributed to language typology. An uncontrolled contrast was therefore not performed; the required design is discussed in Section 5.5 and Section 6.

We compare two attribution methods—the gradient-based Integrated Gradients [15] and the attention-based attention rollout [16]—against random attribution as the random baseline; two further methods, Gradient × Input and leave-one-out occlusion, are evaluated as robustness checks, so that the value-region plausibility results and the eojeol-level faithfulness results in Section 4 compare four attribution methods and the random baseline (rollout under two targets for value-region plausibility). Since the evaluation framework does not depend on any particular attribution method (Section 3.1), other families such as relevance propagation can be evaluated by the same procedure.

The implementation details of the attribution methods are as follows. IG is applied at the text-embedding layer, and its baseline is the [MASK] token embedding rather than the zero vector: the zero vector is an input the model has never observed, reproducing on the attribution side the out-of-distribution trap noted in Section 3.3.2, whereas [MASK] is a token observed repeatedly during pretraining and shares its baseline with the faithfulness perturbation (masking), keeping the two measurements consistent. The integral is approximated by a 32-step Riemann sum; attribution values are summed over the embedding dimension and taken in absolute value; and only the 512 text tokens of the output are used (the 49 visual patch tokens are excluded). Attention rollout averages heads within each layer, incorporates residual connections by adding the identity matrix and re-normalizing rows, and accumulates the layer matrices by multiplication from the input layer upward; its target is reported in two forms: the first token of the ground-truth span, the simplification used in the original protocol, and a span-level target that averages (or sums) the rollout rows of all span tokens; because the sum equals the span length times the mean, the two span-level variants induce identical unit rankings, and IG, whose target is the span probability, already uses a span-level target (Section 4.3). The two robustness-check methods (Section 4.3) are defined as follows: Gradient × Input, the gradient of the span probability with respect to the text embedding multiplied by the embedding and summed over the embedding dimension (absolute value, as for IG), and leave-one-out occlusion, in which each eojeol is replaced by [MASK] in turn, and the resulting drop in the span probability is its attribution score.

The predicted probability of a field span, p_f, is defined as the geometric mean of the probabilities that each token in the span carries the label the model actually predicted: summation systematically favors longer spans and the arithmetic mean hides the collapse of a single token’s probability, whereas the geometric mean matches the span’s all-tokens-must-hold character. Layout jitter translates each bounding box by an integer offset (Δx, Δy) ~ Uniform{−δ,…,+δ} drawn independently per eojeol, clamping coordinates to [0, 1000] (the LayoutXLM normalized coordinate system); the box is translated as a whole, so width and height are preserved except when clamping truncates a box at the boundary, which occurred at a negligible rate even at δ = 134. Coordinate swapping takes, in reading order, the earliest non-gold tokens belonging to another field’s annotated span and exchanges their boxes with the ground-truth tokens one-to-one up to the shorter length; instances with no eligible partner are excluded, and the swap is deterministic with no random draw. Image occlusion maps each eojeol box of the ground-truth span from the 0–1000 coordinate system onto the 224 × 224 pixel grid (corners rounded outward, so partial pixels are occluded) and fills the union of the boxes with zeros. Perturbation randomness (jitter offsets and random attribution) is drawn once per (document, field, condition) from a CPU generator seeded with the training-seed number, so the reported standard deviations combine retraining and perturbation variation; for the primary method, IG, the combined across-seed variability exceeds the document-sampling width (Supplementary Document S3); Section 4.3 reports the seed-paired differences between methods and decomposes the variability into between-seed and between-document components; the retraining and perturbation components themselves are not separated. The attribution and perturbation code is written directly against PyTorch 2.6.0 autograd and the attention matrices, without external interpretability libraries.

Fine-tuning uses a learning rate of 5 × 10⁻⁵, a batch size of 1 with gradient accumulation over 16 steps (effective batch size 16), 3 epochs, and a maximum sequence length of 512, with the 8-bit AdamW optimizer under bf16 mixed precision; all metrics are reported as mean ± standard deviation over three random seeds; two further seeds trained under the identical recipe, and a second backbone (LiLT, a layout-aware model without visual features) fine-tuned under the same recipe with one seed, are reported as robustness checks in Section 4.3 and Supplementary Document S4. Experiments were run on a single NVIDIA GeForce RTX 3060 GPU (8 GB VRAM; NVIDIA, Santa Clara, CA, USA) under Python 3.10.11, PyTorch 2.6.0 (CUDA 12.4), Hugging Face Transformers 4.57.6, and Detectron2 0.6; the pre-trained checkpoint was microsoft/layoutxlm-base from the Hugging Face Hub. The training split contains 5000 documents with adapter-generated weakly supervised labels (Section 5.4), drawn from the corpus after the 300 evaluation documents had been fixed by stratified sampling and their IDs excluded; the two splits share no documents. No validation split was used: the hyperparameters above are defaults adopted without search, so the evaluation split entered no design choice. All code, configurations, and the annotation guideline are released in the public repository (https://github.com/kodocxai/kodocxai, accessed on 30 September 2026). The evaluation code was developed with assistance from Claude (Anthropic; see the GenAI disclosure in the Acknowledgments); all code was reviewed, tested, and verified by the authors, and is released in full for independent validation.

The evaluation sample is selected as follows. Of the 1173 instances carrying a present value, we evaluate 1166, excluding one instance whose span was never specified (a recipient field judged present but wholly unreadable, an exceptional string_na record with no recoverable region and hence an empty span, released as-is in the public annotation layer) and six whose ground-truth eojeols fall outside the 512-token window. The six truncated instances are all issuing authority: in long documents of 173–214 eojeols, the field at the bottom of the page is pushed out of the window; these excluded instances do not enter the reported recall, which therefore overstates the field’s effective coverage on long documents (Section 4.1 and Section 5.5). Absent instances have no ground-truth region and are not evaluated; excluding the one document in which all five fields are absent, 299 documents appear in evaluation. All other string-irrecoverable (string_na) instances are included, since their R_k remains valid (Section 3.4.1). Evaluation is independent of prediction correctness: the attribution and faithfulness target is the label sequence the model actually predicted at the positions of the ground-truth span (when the model predicts no field there, the probability of its predicted O labels is scored), computed once from the unperturbed forward pass and held fixed under every perturbation. A re-aggregation split by prediction correctness is reported in Section 4.3.

4. Results

4.1. Descriptive Statistics of the KoDocXAI Benchmark v1.0

The KoDocXAI benchmark v1.0 annotates 300 permit-and-authorization dispatch documents with value spans and bounding boxes for five fields; it contains 1173 field instances with a present value (1500 slots when the 327 judged absent are included). Two annotators covered 300 and 60 documents (20%), respectively, with median times to first decision of 29 and 21 s per document. The 60 doubly annotated documents are those at every fifth position in the presentation order; the second annotator worked independently with the same guideline and the same automatic drafts. Disagreements were resolved by discussion between the two annotators.

Cohen’s κ for field presence was 1.000 for every field. This value is not degenerate: in the 60 doubly annotated documents, both the present and absent classes occur for every field (e.g., document number: 36 present, 24 absent), and there were zero presence disagreements. “Status agreement” is the simple agreement rate over the three values present/absent/string-irrecoverable, capturing the reproducibility of the string_na judgment, which κ does not distinguish. Span Jaccard is the intersection over union of the eojeol sets selected by the two annotators, computed per document–field instance judged present by both and then averaged (macro average); it indicates the stability of the value-region rationale reference R_k.

The residual span disagreements concentrate in string-irrecoverable cases. All three span disagreements for the issuance date (exact match 91.7%) occurred in documents that both annotators judged string-irrecoverable: when a value is only partially observable due to seal overlap or line splits, the annotators differed on whether to include residual fragment eojeols in the span. Because such cases are excluded from string match scoring, they do not surface in the string match rate (100%).

Absence rates diverge sharply across fields. Title and recipient exist in nearly every document (6.0% and 1.7% absent), whereas document number and issuance date are absent in 34.3% of documents. This is because the sample includes continuation pages of multi-page documents and internally approved drafts, whose header form fields are printed only on the first page. Since absence arises structurally, how absent fields are handled affects the evaluation metrics (Section 3.4.1).

String-irrecoverable cases are concentrated in the issuance date and the issuing authority (43 and 45 cases, respectively). For the issuing authority, most cases arise when the official seal covers a character so that its eojeol is missing from the OCR word list; this is the systematic formatting–convention loss described in Section 3.4.1.

With the ground-truth annotations in place, the accuracy of the automatic drafts produced by the label adapter can also be computed as a baseline. Per-field end-to-end accuracy (the proportion of documents with a present value for which the adapter also produced the correct value; string-irrecoverable instances are excluded from the denominator, as in Table 2) was 98.2% for title, 74.1% for recipient, 79.0% for document number, 75.3% for issuance date, and 89.2% for issuing authority; detection precision was 100.0% for all fields, and detection recall ranged from 71.3% (issuing authority) to 99.0% (document number). Because the same adapter generates the weakly supervised labels of the training split, this baseline serves as a reference for gauging training-label quality (Section 5.4).

How well the fine-tuned model performs the task itself is the basis for interpreting the explanation evaluation. Table 3 reports eojeol-level precision, recall, and F1, together with exact span match (EM) on the 300-document evaluation split, averaged over three seeds. The micro F1 is 0.923 ± 0.002, indicating that the model attains substantial performance on the task. Because absent fields are excluded, these values are conditional extraction performance on gold-present fields, not end-to-end KIE scores. Section 4.3 additionally examines the explanation metrics conditioned on prediction correctness. Among the fields, the asymmetry of the issuing authority stands out, with precision 0.986 against recall 0.776: when it points, it points correctly, but it misses often. The reported recall counts only the evaluated sample and is dominated by the training-label coverage deficit (Section 5.4); six further instances, truncated beyond the 512-token window, are excluded from the sample altogether (Section 3.5), so the field’s effective coverage on long documents is lower than the reported figure.

Table 3. Per-field KIE performance of fine-tuned LayoutXLM-base. Evaluation split of 300 documents, eojeol level, and mean ± standard deviation over three seeds. EM is the proportion of predicted spans exactly matching the ground truth. Scores are conditional on gold-present fields: absent fields are skipped, so spurious predictions on them are not counted against precision; n excludes the six truncated instances and the one empty-span instance (Section 3.5).

Field	Precision	Recall	F1	EM	n
Title	0.936 ± 0.007	0.976 ± 0.011	0.956 ± 0.002	0.775 ± 0.002	282
Recipient	0.918 ± 0.010	0.870 ± 0.017	0.893 ± 0.006	0.635 ± 0.014	294
Document number	0.967 ± 0.000	0.974 ± 0.008	0.971 ± 0.004	0.863 ± 0.018	197
Issuance date	0.934 ± 0.003	0.922 ± 0.002	0.928 ± 0.002	0.692 ± 0.009	197
Issuing authority	0.986 ± 0.007	0.776 ± 0.009	0.868 ± 0.003	0.696 ± 0.006	196
Overall (micro)	0.943 ± 0.004	0.903 ± 0.007	0.923 ± 0.002	-	1166

The Effect of Field Definition Ambiguity on Agreement

During annotation, it emerged that the definition of the issuing authority field failed to cover one case. With no rule distinguishing the agency name letterhead printed at the top of a dispatch document (“김 해 시”, Gimhae City) from the head-of-agency signature at the bottom (“김해시장”, Mayor of Gimhae), the letterhead was selected as the value in documents lacking a bottom signature (75 of 300 documents for the first annotator; among the 60 doubly annotated documents, 16 fall in this category, and in 15 of them, the two annotators’ selections coincided). Table 4 reports agreement before and after the rule was made explicit and the affected documents were re-adjudicated.

Table 4. Inter-annotator agreement before and after clarifying the definition of the issuing authority field. The other four fields are omitted as they showed no change.

Stage	Status Agr.	Exact Span	Span Jaccard
Before clarification	93.3%	98.1%	0.981
After clarification	96.7%	100.0%	1.000

Agreement was already high before the clarification. When two annotators share the same misjudgment, agreement rises rather than falls, so high IAA does not guarantee that the annotation rules are valid. This case is a counterexample to the practice of presenting IAA as a sufficient condition for annotation quality, and it suggests that benchmark construction should verify the completeness of field definitions separately from reporting agreement. We accordingly report both pre- and post-adjudication values. Computing IAA on pre-adjudication annotations is the principle; when the adjudication stems from a rule revision; however, the post-revision value is what reflects the quality of the released ground truth.

4.2. Effect of the Aggregation Level on Faithfulness (RQ1)

Table 5 reports COMP@10, SUFF@10, and AOPC measured at the three aggregation levels (subword, morpheme, eojeol) and, at the morpheme level, under the three masking conditions (all, content, functional); Figure 3 shows AOPC across the aggregation levels. The masking conditions are split only at the morpheme level, and mixed morphemes are excluded from both, so content and functional do not sum to all (Section 3.2). Table 5 also lists, at the eojeol level, the two additional attribution methods evaluated as robustness checks; they are discussed in Section 4.3.

Table 5. Faithfulness by aggregation level and masking condition: comprehensiveness (COMP@10), sufficiency (SUFF@10), and the area over the perturbation curve (AOPC; unweighted mean of COMP@k over the grid k ∈ {1, 3, 5, 10}; a discrete variant, not an integrated area), mean ± standard deviation over three seeds. Masking conditions (all, content, functional) apply only at the morpheme level; mixed morphemes are excluded from both content and functional (Section 3.2). Higher is better for COMP and AOPC; lower is better for SUFF. At the eojeol level, the table also reports the two additional attribution methods: Gradient × Input and leave-one-out occlusion (Section 3.5); rollout is shown under the first-token target of the original protocol (Section 4.3).

Level	Masking	Method	COMP@10	SUFF@10	AOPC
Subword	all	IG	0.6676 ± 0.0762	0.0622 ± 0.0348	0.6101 ± 0.0989
Subword	all	rollout	0.0335 ± 0.0055	0.7283 ± 0.0108	0.0147 ± 0.0021
Subword	all	random baseline	0.0428 ± 0.0115	0.6982 ± 0.0139	0.0209 ± 0.0061
Morpheme	all	IG	0.6668 ± 0.0769	0.0602 ± 0.0338	0.6093 ± 0.0986
Morpheme	all	rollout	0.0368 ± 0.0107	0.7268 ± 0.0052	0.0152 ± 0.0041
Morpheme	all	random baseline	0.0260 ± 0.0074	0.7278 ± 0.0116	0.0105 ± 0.0027
Morpheme	content	IG	0.6663 ± 0.0762	0.0601 ± 0.0337	0.6090 ± 0.0981
Morpheme	content	rollout	0.0399 ± 0.0126	0.7231 ± 0.0069	0.0161 ± 0.0048
Morpheme	content	random baseline	0.0259 ± 0.0073	0.7273 ± 0.0120	0.0105 ± 0.0027
Morpheme	functional	IG	0.0096 ± 0.0033	0.7288 ± 0.0297	0.0056 ± 0.0019
Morpheme	functional	rollout	0.0078 ± 0.0027	0.7338 ± 0.0243	0.0037 ± 0.0013
Morpheme	functional	random baseline	0.0080 ± 0.0031	0.7335 ± 0.0239	0.0039 ± 0.0014
Eojeol	all	IG	0.6671 ± 0.0753	0.0562 ± 0.0317	0.6086 ± 0.0969
Eojeol	all	Gradient × Input	0.6169 ± 0.0691	0.0937 ± 0.0341	0.5110 ± 0.0735
Eojeol	all	leave-one-out occlusion	0.6490 ± 0.0730	0.0525 ± 0.0200	0.6027 ± 0.0844
Eojeol	all	rollout	0.0235 ± 0.0053	0.7404 ± 0.0100	0.0113 ± 0.0020
Eojeol	all	random baseline	0.0225 ± 0.0066	0.7382 ± 0.0127	0.0113 ± 0.0035

Figure 3. AOPC by aggregation level and masking condition (subword/morpheme all, content, functional/eojeol) for IG, rollout, and the random baseline, averaged over three seeds with standard deviation error bars. IG is practically identical across the four combinations outside the functional-only condition, and rollout and the random baseline overlap throughout; the aggregation level does not change the outcome of the faithfulness evaluation (Table 5).

The aggregation level did not change the outcome of the faithfulness evaluation. The AOPC of IG is 0.610 at the subword level, 0.609 at the morpheme level, and 0.609 at the eojeol level: the differences across levels (at most 0.002) are roughly 2% of the across-seed standard deviation (0.097–0.099). Attention rollout and the random baseline overlap within 0.010–0.021 at every level and are indistinguishable from each other. The ranking of methods (IG ≫ rollout ≈ random) is likewise identical across the three levels.

Across the masking conditions, the content-only condition was practically identical to the all condition (IG AOPC 0.609 vs. 0.609), whereas masking the eojeols selected through functional morphemes only collapsed all three methods evaluated at that level to 0.004–0.006, erasing the differences between methods; under this masking protocol, the evidence reached through particles and endings made little measurable contribution to prediction. This accords with the noun-phrase-dominant character of the corpus, whose functional-morpheme ratio is only 9.0% (Section 3.2); the implications are discussed in Section 5.1.

Two robustness checks support this reading. First, on a fixed subset of 100 documents (identifiers released with the code) and all three seeds, we repeated the eojeol-level faithfulness evaluation with two alternative text perturbations, token deletion and random in-vocabulary substitution, with and without span protection (Supplementary Document S4). The ranking IG ≫ rollout ≈ random holds under every strategy (protected COMP@10: IG 0.68–0.77 vs. 0.02–0.17 for the other two), and protection lowers the scores under all three strategies (unprotected minus protected COMP@10 for IG: 0.21 under [MASK] replacement, 0.18 under deletion, 0.15 under substitution). Deletion also raises the scores of the random baseline (COMP@10 0.15 protected, 0.27 unprotected), a response to anomalous input that supports the choice of the pretraining [MASK] token (Section 3.3.2). Second, matching the perturbation budget by the share of non-protected characters (5–30%) instead of a fixed top-k leaves the aggregation-level conclusion intact (at a fixed k, coarser units remove more text because aggregation to coarser units tends to rank longer eojeols higher; the masked unit is always the whole eojeol): the spread of IG’s COMP across subword, morpheme, and eojeol stays within 0.010 at every budget, about one-eighth of the seed-level standard deviation; the two low-signal methods vary more across levels under matched budgets but remain far below IG at every budget (Supplementary Document S4).

In the setting without protection of the ground-truth span, by contrast, the aggregation level appears to change the faithfulness of rollout by more than a factor of two (Appendix A). Since the two settings differ only in whether the span is protected, this apparent effect is isolated as an artifact of self-evidence masking. The interpretation is given in Section 5.1.

4.3. Value-Region Plausibility by Attribution Method (RQ2)

Table 6 reports the spatial agreement between the attributions re-aggregated at the eojeol level and the value-region rationale reference R_k, measured by IoU@1, IoU@5, F1@5, and Hit@1. In documents carrying multiple dispatch copies on one page, the eojeols of the lower drafts are ignored before computation (Section 3.4.1).

Table 6. Value-region plausibility by attribution method: spatial agreement with the value-region rationale reference R_k at the eojeol level, mean ± standard deviation over three seeds. In documents carrying multiple dispatch copies, the eojeols of lower drafts are ignored (Section 3.4.1). IoU denotes intersection over union. Rollout is reported under the first-token target of the original protocol and under a consistent span-level target (mean of the rollout rows over all span tokens; the sum variant induces the same ranking). Gradient × Input and leave-one-out occlusion are the two additional attribution methods evaluated as robustness checks (Section 3.5 and Section 4.3).

Method	IoU@1	IoU@5	F1@5	Hit@1
IG	0.175 ± 0.102	0.479 ± 0.030	0.597 ± 0.021	0.402 ± 0.214
Gradient × Input	0.329 ± 0.019	0.383 ± 0.024	0.504 ± 0.025	0.684 ± 0.054
Leave-one-out occlusion	0.151 ± 0.049	0.455 ± 0.007	0.566 ± 0.009	0.345 ± 0.094
Rollout (first-token target)	0.000 ± 0.000	0.045 ± 0.002	0.071 ± 0.004	0.000 ± 0.000
Rollout (span-level target)	0.000 ± 0.000	0.053 ± 0.001	0.083 ± 0.004	0.000 ± 0.000
Random baseline	0.015 ± 0.001	0.028 ± 0.001	0.038 ± 0.001	0.024 ± 0.002

IG exceeded the random baseline by a wide margin on every metric (IoU@5 0.479 vs. 0.028; Hit@1 0.402 vs. 0.024). In contrast, attention rollout scored 0.000 on Hit@1 and IoU@1 in every seed—below even the random baseline—whereas its IoU@5 (0.045) and F1@5 (0.071) exceed the baseline (0.028 and 0.038) only by small absolute margins. Re-evaluating rollout against the span-level target changes this picture only marginally (IoU@5 0.053, F1@5 0.083; Hit@1 and IoU@1 remain 0.000), and its eojeol-level faithfulness stays at the random baseline level under both targets (AOPC 0.0113): the weakness of rollout is a property of the method in this setting, not of the first-token simplification. Since rollout also overlapped with the random baseline on faithfulness (Table 5), it shows no meaningful advantage over that baseline on either evaluation axis.

Splitting by prediction correctness leaves this conclusion intact. Re-aggregating, within each seed, only the instances whose predicted span exactly matches the ground truth (848 on average, 72.7%) gives IG an IoU@5 of 0.485 ± 0.029 and an AOPC of 0.656 ± 0.094—equal to or higher than the overall means—while the random baseline remains at 0.022 ± 0.002 (IoU@5) and 0.013 ± 0.004 (AOPC): the conclusions of Table 5 and Table 6 do not rest on mispredicted cases. Rollout’s IoU@5, by contrast, rises on incorrect predictions (correct 0.037 ± 0.002 vs. incorrect 0.066 ± 0.001), and the random baseline moves in the same direction (correct 0.022 ± 0.002 vs. incorrect 0.042 ± 0.001), suggesting that the pattern reflects a baseline-level fluctuation, rather than genuine alignment with the rationale.

The ranking is also stable across seeds. For every metric of Table 5 and Table 6, the seed-paired differences IG—random and IG—rollout, computed within each fine-tuning seed, have the same sign in all three seeds, and the document-level cluster bootstrap 95% intervals exclude zero in all 14 contrasts (e.g., Hit@1 IG—random ranges from +0.195 to +0.681 across seeds, interval [+0.362, +0.393]); between-document variability exceeds between-seed variability for IG on every metric (Supplementary Document S4). Absolute scores vary across seeds, but the ranking on which the conclusions rest does not.

The ranking survives three further checks (the first in Table 5 and Table 6; all three in Supplementary Document S4). First, the two additional attribution methods evaluated with the identical protocol, Gradient × Input and leave-one-out occlusion (their rows in Table 5 and Table 6), both outperform rollout and the random baseline on every metric (Hit@1 0.684 and 0.345; COMP@10 0.617 and 0.649; AOPC 0.511 and 0.603), so the discrimination of the reference is not specific to IG. The methods differ in where they are strongest: Gradient × Input gives the sharpest top-1 pointing (Hit@1 0.684 and IoU@1 0.329, against 0.402 and 0.175 for IG), whereas IG keeps the best top-5 coverage and the highest comprehensiveness and AOPC (IoU@5 0.479 against 0.383; AOPC 0.609 against 0.511), which is itself an argument for reporting the axes and the top-k levels separately (Section 3.3.1 and Section 3.3.5); occlusion shares its perturbation primitive with the faithfulness protocol, so its faithfulness scores carry a circularity caveat. Second, two additional training seeds under the identical recipe (five in total; micro F1 0.921 and 0.924) leave every paired contrast with the same sign in all five seeds, with bootstrap intervals excluding zero; the main tables keep the original three seeds. Third, replacing the backbone by LiLT, a layout-aware model without visual features fine-tuned under the same recipe (one seed; micro F1 0.911), reproduces the sign of all 14 contrasts (IG Hit@1 0.835 against 0.021 for rollout and 0.020 for the random baseline), while absolute scores shift with the model (IG COMP@10 0.479 against 0.667 on LayoutXLM): the ranking on which the conclusions rest is not tied to the LayoutXLM architecture, although a model in which visual features matter remains untested (Section 6).

4.4. Field-Wise and Modality-Wise Perturbation Sensitivity (RQ3)

Table 7 reports, for each field, the change in prediction probability under text perturbation in three settings (masking the IG top-10 context units with the ground-truth span protected—COMP@10 of Section 3.3.2—masking the ground-truth value span itself, and masking both), layout perturbation (three levels of coordinate jitter on the ground-truth region, and swapping coordinates between fields), and image perturbation (occluding patches of the ground-truth region). Following the principle of Section 3.3.3, context masking removes context while preserving the identity of the value; the two value-masking settings, added to complete the modality picture, also remove the value text, whereas layout and image perturbations target the ground-truth region directly. The jitter magnitude is set to δ ∈ {25, 67, 134}—integer values equal to 0.37×, 1.00×, and 2.01× the standard deviation of the natural coordinate variation measured on the corpus (66.8), in the LayoutXLM normalized coordinate system (0–1000).

Table 7. Field-wise and modality-wise perturbation sensitivity. Each value is the change in prediction probability under perturbation (mean ± standard deviation over three seeds); larger values indicate stronger dependence on the modality. Text columns (all computed at the eojeol level): [MASK] masking of the IG top-10 context eojeols (ground-truth value span protected; COMP@10), of the ground-truth value span itself, and of both together; layout: jitter at three levels and coordinate swapping between fields; image: patch occlusion of the ground-truth region. Text (value) is attribution-independent, while Text (context) masks IG-selected units and is thus attribution-conditioned (it coincides with IG’s eojeol-level COMP@10); the text columns are read here as modality sensitivity, not as attribution faithfulness (Section 3.3.2).

Field	Text (Context)	Text (Value)	Text (Value + Context)	Jitter δ = 25	Jitter δ = 67	Jitter δ = 134	Coordinate Swap	Image
Title	0.7352 ± 0.1390	0.5511 ± 0.1205	0.9648 ± 0.0121	0.0019 ± 0.0003	0.0179 ± 0.0054	0.1204 ± 0.0123	0.3828 ± 0.0269	0.0000 ± 0.0000
Recipient	0.7599 ± 0.1061	0.3922 ± 0.2154	0.8470 ± 0.0391	0.0022 ± 0.0004	0.0280 ± 0.0173	0.1592 ± 0.0668	0.3341 ± 0.0987	0.0000 ± 0.0000
Document number	0.9565 ± 0.0088	0.2074 ± 0.0640	0.9492 ± 0.0189	0.0028 ± 0.0013	0.0247 ± 0.0012	0.1664 ± 0.0585	0.2487 ± 0.0633	0.0000 ± 0.0000
Issuance date	0.7938 ± 0.1936	0.5387 ± 0.0833	0.9319 ± 0.0122	0.0022 ± 0.0005	0.0266 ± 0.0076	0.2057 ± 0.0704	0.2342 ± 0.0459	0.0000 ± 0.0000
Issuing authority	0.0116 ± 0.0155	0.6940 ± 0.0070	0.6986 ± 0.0078	0.0007 ± 0.0003	0.0042 ± 0.0044	0.0139 ± 0.0063	0.2080 ± 0.1030	0.0000 ± 0.0000

Context masking alone produced the largest change in most fields (title 0.735, recipient 0.760, document number 0.957, issuance date 0.794), and masking value and context together lower the probability by 0.882 on average (instance-weighted), close to a collapse. The interventions differ in scope across modalities, so this is not a like-for-like comparison, but under the present interventions the combined text perturbation produced by far the largest drop (layout at most 0.383; image zero). The fields differ sharply in the evidence they rely on: the document number depends almost entirely on context (0.957 context-only vs. 0.207 value-only, consistent with reliance on IG-selected contextual cues such as the printed item label), the title and issuance date on both. The exception is the issuing authority, whose response to context masking is only 0.012, one-60th to one-80th of the other fields; this field is predicted with little reliance on text outside the value span. Masking its value span, by contrast, lowers the probability by 0.694 (Table 7), so the field is read from the value text and its position rather than from context. It also responds to swapping coordinates between fields at 0.208, and when seal-overlapped and normal documents are aggregated separately, this response concentrates in the normal documents (0.268 vs. 0.008; Section 5.2). At least in documents whose text is intact, the issuing authority appears to be decided from the value text and its position.

Within the layout modality, the response to jitter increases monotonically with δ: negligible at δ = 25 (0.37× the natural variation; 0.001–0.003), 0.004–0.028 at δ = 67, and 0.014–0.206 at δ = 134. Swapping coordinates between fields, at 0.208–0.383, exceeds every jitter level. The effect of image perturbation was 0.000 in all three seeds. The character of the layout dependence and the inspection and interpretation of the image modality inactivity are treated in Section 5.3.

4.5. Divergence Between Value-Region Plausibility and Faithfulness

Value-region plausibility and faithfulness are independent axes (Section 3.3.5), and cases where the two diverge are in fact observed. Figure 4 shows two contrasting cases at the eojeol level with IG on seed 1. The two cases are illustrations of a divergence that Table 5 and Table 6 establish on all evaluated instances and seeds; they are not themselves the evidence for it.

Figure 4. Two cases of value-region plausibility–faithfulness divergence (seed 1, IG, eojeol level). (a) Issuing authority: The attribution points at the ground-truth region (IoU@5 0.911), but faithfulness is negative (AOPC −0.278). (b) Issuance date: None of the top-5 attributed units belong to the ground-truth value (IoU@5 0.000), yet masking the top-ranked units collapses the prediction at every k (AOPC 0.995). Faithfulness masking excludes the protected ground-truth span before selecting the top-k, so in (a) the masked units differ from the displayed top-5, and the case reads as a modality–profile contrast, whereas in (b) the displayed top-5 and the set masked at k = 5 coincide (Section 3.3.2). Personally identifiable information (staff names, phone/fax numbers, e-mail addresses, the private recipient’s name and address, and handwritten signatures) is redacted; the recipient in (a) is a public institution and is left intact, as are the annotated fields and official seals, and the redaction covers none of the top-k attributed eojeols or the value-region rationale reference R_k.

(a) High value-region plausibility, negative faithfulness: Issuing authority (document 5350109-1999-0002-1246). The attribution points precisely at the ground-truth region (IoU@5 0.911), yet the AOPC is negative (−0.278): the predicted probability does not fall but rises. Two observations are needed to read this case correctly. First, faithfulness masking excludes the protected ground-truth span before selecting the top-k (Section 3.3.2), so the units actually masked are the next-ranked non-gold units, not the displayed top-5; the value span itself was never masked. Second, this is the field whose context sensitivity collapses to 0.012 while its value sensitivity is 0.694 (Table 7): the model consults the value text and its position, not the surrounding context that those masked units represent. The divergence thus reflects not a failure of attribution but the field’s modality profile (Section 5.4). By protocol design, the two metrics in (a) are computed on different unit sets; case (b) below, where the displayed top-5 and the set masked at k = 5 coincide, provides the same-set divergence.

(b) Zero value-region plausibility, high faithfulness: Issuance date (document 5350109-2000-0001-0727). The top-5 attributed units are the header labels “시행일자” (issuance date) and “문서번호” (document number), a parenthesis, a fragment of the document number, and a URL. None of them are part of the ground-truth value “2000. 3. 22” (IoU@5 0.000), yet masking its top-ranked units collapses the prediction at every k of the grid (AOPC 0.995, hence COMP@5 ≥ 0.98). The model solves the field not from the date value itself, but from the header labels and their positional relations. Under the value-region definition of Section 3.1, this is not a failure of the metric but a case the metric is not designed to score: the explanation is faithful to evidence outside the value region, which the context sensitivity column of Table 7 captures.

The two cases differ in kind: (a) is a matter of modality dependence, whereas (b) is a case where the evidence the model actually uses differs from the value-region reference. Neither would surface under single-axis reporting, and the contrast in (b) in particular would have been buried under the trivial “masking the answer breaks the prediction” had the context perturbation not protected the value span (Section 3.3.2).

5. Discussion

5.1. The Aggregation-Level Effect and Its Conditions (RQ1)

The aggregation level did not materially change the faithfulness scores or the ranking of methods (Section 4.2, Table 5), answering RQ1 in the negative. The unit mismatch itself is real—6.3% of subwords cross a morpheme boundary (Section 3.2)—but masking the eojeols selected through functional morphemes only collapses all three methods evaluated at that level onto the random baseline level (Table 5), so the evidence reached through particles and endings contributed little that the masking could measure. The risk of merging units lies in summing heterogeneous contributions into one value; when one side contributes almost nothing, there is nothing for the summation to distort. The null result is consistent with, and may reflect, the noun-phrase-dominant character of the corpus, and unit effects may still appear in documents with more running text (Section 6).

Meanwhile, we measured that something that looks like an aggregation-level effect can be manufactured by a measurement flaw: without protection of the ground-truth span, the faithfulness of rollout appears to change by more than a factor of two across aggregation levels (Appendix A)—a self-evidence effect in which coarser units place more ground-truth eojeols in the top k and mask them. An aggregation-level effect, therefore, can be meaningfully observed only (i) in corpora where functional morphemes actually contribute, and (ii) under a protocol in which self-evidence is controlled.

5.2. Value-Region Rationale Reference Derived from KIE Annotations: Conditions and Limits (RQ2)

The value-region rationale reference R_k derived from KIE annotations discriminated the value-region plausibility of the methods without dedicated rationale annotation, answering RQ2 in the affirmative (Section 4.3, Table 6). Rollout scored 0.000 on Hit@1 and IoU@1 in every seed, below even the random baseline; in all 10 documents inspected, its top unit was the first eojeol of the document, unrelated to the ground truth—a positional bias of the layer-wise attention accumulation toward early sequence positions (reproducible with check_attribution_topk.py in the released code). The span-level target of Section 4.3 leaves these scores unchanged, so the low scores are attributable to the method. On the faithfulness axis, rollout was likewise indistinguishable from random, and without span protection the faithfulness of uninformative methods was inflated far more than IG’s (rollout +62%, random +43% vs. IG +19%; Appendix A). Without the random baseline and the protection protocol, a widely used attention summary method would have been reported with a plausible-looking explanatory score.

The derivation is valid when the field value exists locally at one place on the document, and two exceptions were observed. First, because of the “one draft, multiple dispatches” practice, a single page carries several dispatch copies (23.0% of the sample); we took the topmost draft as the ground truth and ignored the rest in evaluation (Section 3.4.1)—a sidestep of the multi-instance problem whose solution requires per-instance annotation and multi-instance matching metrics (Section 6). Second, 45 issuing authority instances whose string cannot be recovered because of seal overlap (Table 2) were kept in the value-region plausibility sample: the reference region remains intact even where the string is unrecoverable, so the value-region reference is valid over a wider range than the string ground truth. This loss is also a naturally occurring damage to the text modality—splitting the field into normal and seal-overlapped documents separates its coordinate-swapping response (0.268 vs. 0.008), showing that the positional dependence of Section 4.4 is a property of documents whose text is intact. Since seal overlap is source data variation rather than random assignment, this contrast must be read as an association, and the small string_na subsample (45 instances) remains a limitation.

Because the reference is the value region, its extent is a modeling choice, and we probed its influence by dilating every reference box by 10%, 25%, and 50% on all sides and absorbing the neighboring eojeols whose centers fall inside (the source annotations contain no field-label boxes, so a label-inclusive definition could not be tested directly). The ranking of methods is unchanged at every dilation: IG rises moderately, as near-miss pointings are reclassified as hits (Hit@1 0.401 → 0.440 at 25% and 0.595 at 50%; the 0% value differs from Table 6 by a single top-1 tie), while rollout stays at the baseline level (Hit@1 at most 0.002) and the random baseline moves by at most 0.004 (Supplementary Document S4). Two consequences follow for reading the results. First, because the reference is defined as the value region and has not been validated against human rationales, the value-region plausibility scores are conditional on that definition: an explanation that faithfully relies on a field label or on context is scored low on this axis even when its reliance on that evidence is confirmed on the faithfulness axis (Figure 4b). Second, the absolute values depend on the extent of the reference, in the direction and by the amount shown by the dilation (upward, by about 0.19 in the Hit@1 of IG at 50% dilation), so they should not be compared with plausibility scores obtained against human rationales elsewhere. The ordering of methods, on which the conclusions rest, was unchanged under every dilation tested. Validating the reference against human rationales on a subset remains future work.

5.3. Decomposed Modality Sensitivities (RQ3)

Prediction sensitivity to the image modality is practically zero: the change in prediction probability under reference region occlusion was 0.000 for every field in all three seeds (Table 7). Three checks confirmed that this was not a failure to apply the perturbation—replacing the entire image with zeros or random noise changed the logits only at the 10⁻⁴–10⁻³ level (299 evaluation documents, three seeds); measuring on logits rather than probabilities rules out softmax saturation; and the changes are not exactly zero. The small but nonzero logit changes make a complete disconnection of the image pathway unlikely, although they do not by themselves rule out implementation defects (reproducible with check_image_modality.py). On the same logit scale, the whole-image perturbation remains at least two orders of magnitude below both the text and the layout figures in every seed (a statement about the size of the image response, not a like-for-like comparison of interventions; Section 4.4). We hypothesize that this may reflect low character legibility at the 224 × 224 input resolution, combined with the task information already present in the OCR text and coordinates—an architecture classified as multimodal effectively operates with two modalities on this task, and a single summed faithfulness index would have misread the image share as text and layout contributions (the modality-wise reporting principle of Section 3.3.3). That the framework can certify a modality as inert, rather than presuming that every modality contributes, is precisely what modality-wise reporting is for; the multimodal claim of this study is accordingly limited to the channels the task exercises, text and layout, and a known-groups validation with a model in which visual features demonstrably matter is the next step (Section 6). Consistent with this reading, a backbone without visual input (LiLT, text and layout only; Section 4.3) reaches micro F1 0.911 against 0.923 for LayoutXLM on the same task, although this does not replace a known-groups test with a model in which visual features matter.

The layout modality barely reacts to jitter at about 0.4× the natural variation (δ = 25; 0.001–0.003) but reacts clearly at about 2× (δ = 134) and most strongly to swapping coordinates between fields (0.208–0.383; Table 7)—a pattern consistent with dependence on relative arrangement rather than absolute coordinates (an interpretation, not a demonstrated separation). This is a task-appropriate outcome of learning on a regular form, but in irregular documents the same dependence could cause misclassification (Section 5.5). Note also that LayoutXLM appends 7 × 7 = 49 visual patch tokens after the text tokens; unless this segment is separated during re-aggregation, text and image contributions blend—an implementation condition that any study claiming modality-wise decomposition needs to state.

5.4. Methodological Observations from Benchmark Construction

On the issuing authority field, the misjudgment of selecting the top letterhead as the value appeared in 75 of 300 documents for the first annotator, and in 15 of the 60 doubly annotated documents the two annotators’ selections coincided—yet the exact span match before the definition was clarified was already 98.1% (Table 4). Two annotators reading the same guideline and receiving the same automatic drafts can deviate in the same direction on cases the guideline fails to regulate, and agreement then rises instead of falling: inter-annotator agreement does not guarantee the validity of annotation rules. We accordingly recommend conducting a completeness check of field definitions (absent values, multiple occurrences, partially observable values) separately from computing agreement, and reporting agreement both before and after any rule revision. The concern that the automatic drafts anchored the annotation was narrowed by direct measurement (71.9% acceptance; the 58 draft-less spans are structurally immune) but cannot be excluded entirely, and we state it as a limitation.

The training split consists of weakly supervised adapter labels; human-verified ground truth exists only in the evaluation split. The issuing authority is at once the field with the noisiest training labels (assigned in 35.6% of training documents vs. 67.3% prevalence in the verified pilot; adapter detection recall 71.3%, the lowest of the five fields—Section 4.1) and the field with the most singular dependence structure: the model barely responds to context masking yet responds to value masking and coordinate swapping (Table 7). In a field whose textual signal was not consistently present during training, the model appears to have learned to rely on the value text and its stable position instead of surrounding context (an association-level observation). Whether low performance on a field reflects the model or label noise should be judged against the adapter baseline of Section 4.1.

5.5. Threats to Validity

Scale. The KoDocXAI benchmark v1.0 is a 300-document pilot. Document-level cluster bootstrapping (Supplementary Document S3) shows that the three conclusions of this study hold in all 500 resamples from n = 150 at the latest, and that the main source of remaining uncertainty is not the number of documents but seed-associated variability (retraining combined with perturbation randomness; Section 3.5). The main tables are based on three seeds. The seed-paired analysis of Section 4.3 bounds the concern for the ranking: all 14 method contrasts keep their sign in every one of five training seeds, with bootstrap intervals excluding zero, and between-document variability exceeds between-seed variability (a statement about per-document dispersion; the uncertainty of the corpus mean, discussed above, is governed by the few seeds rather than by the 300 documents). Five seeds remain insufficient for fine-grained ranking of methods whose scores are close, and the structural properties of the corpus set out under the surface-form definition below mean that the reported numbers characterize this corpus rather than serving as universal reference values.

Single form type, institution, and model. The target is one document type—the permit-and-authorization dispatch document—produced mostly by a single local government, and the main experiments used a single model, LayoutXLM-base (Section 3.5); a second backbone without visual features (text and layout only) reproduced the method ranking under one seed (Section 4.3), but no model in which visual features matter has been evaluated. The regularity of the form is both the condition that made automatic derivation of the value-region rationale reference possible and a constraint on generalization, and whether the observed modality dependence structure is specific to this model or intrinsic to the task can be separated only by evaluating multiple models under the same protocol (with NAOPC to correct scale differences across models; Section 3.3.2).

Measurement settings. The rollout target of the original protocol was the first token of the ground-truth span; re-evaluation against a span-level target (Section 4.3) leaves rollout at the baseline level, so the low scores are attributable to the method rather than to the target. More generally, the faithfulness and modality sensitivity findings are conditional on the operational definitions of the perturbations—[MASK] replacement for text, jitter calibrated to natural coordinate variation for layout, and patch occlusion for image—and none of these is shown to be equivalent to real-world document variation. The alternative-perturbation check of Section 4.2 bounds this concern for the text channel only. Truncation at the maximum sequence length of 512 removed the issuing authority from evaluation in six long documents, so the reported recall overstates the field’s effective coverage on long documents (Section 4.1).

Surface-form definition of field values. In benchmark v1.0, a field value is the span written in the corresponding form field, which fixes the extraction target reproducibly (Section 3.4). This definition departs from the answer a user might expect at three measurable points. For the recipient field, 86 of the 294 evaluated recipient instances (29.3%; Table 3) are routing notations rather than recipient names—“see the recipient list” (“수신처 참조” and its variants; 66) and “internal approval” (“내부결재”; 20)—so the extracted span does not itself name a recipient. In 132 of the 1173 present instances (11.3%), the region is identified, but the string cannot be recovered (string_na; Section 3.4.1). Furthermore, on 23.0% of the pages, the same field occurs in several dispatch copies, of which one is taken as the ground truth (Section 5.2). The explanation metrics of this study are defined on the surface span and are unaffected, but the benchmark measures surface extraction, not the resolution of what a span denotes; these are structural properties of the corpus that bound the interpretation of the extraction scores.

Temporal bias. The sample consists of documents from 1994 to 2002, whose scan quality is lower and OCR misread rate higher than in recent documents. Low-quality scans and seal overlap naturally provide the conditions under which modality-wise evaluation is informative (Section 5.2), but whether our numerical findings replicate on contemporary electronic documents cannot be confirmed with this data alone (Section 6).

Absence of a cross-lingual contrast. The problem setting—that agglutinative morphology gives rise to the aggregation-level issue—would be supported by a contrast with a weakly inflecting language, but in currently available configurations (e.g., LayoutLMv3 + CORD [11]), the model, pretraining language, corpus, and document form change simultaneously, so the observed differences cannot be attributed to language typology. This paper accordingly makes no cross-lingual generalization claim and confines its findings to Korean form documents (the contrast design is left to Section 6).

Benchmark accessibility. Under the source data’s terms of use, the originals cannot be redistributed, and applications to AI-Hub are limited to residents of the Republic of Korea, so despite the released annotation layer and deterministic reassembly scripts, full reconstruction of the benchmark is domestically restricted.

Pilot study application of the qualitative layer. The rubric of Section 3.3.4 was applied once, under the protocol of Supplementary Document S2, except for the rater condition noted below, to 50 explanations (five fields × 10; IG and attention rollout 25 each; 35 correct and 15 incorrect predictions under the scoring rule of Section 3.4.1) by three raters outside the author team, with the method identity and answer key withheld, scoring independently after 10 practice items (the first author also scored all items; the secondary analysis is reported in Supplementary Document S2 only). The raters were a computer science professor outside document AI, a graduate student with experience in processing contract documents, and the second annotator of the benchmark; none is a practitioner of official document processing or a document AI researcher, so the rater condition of the protocol is not met and the pilot study is reported as a feasibility check, not as validation. The inter-rater reliability (Krippendorff’s ordinal α; pre-specified threshold 0.667) was 0.893 (bootstrap 95% CI 0.818–0.936) for user verifiability and 0.977 (0.951–0.993) for semantic consistency. Verifiability scores separated the two methods (median of per-item rater means 4.00 for IG against 1.67 for rollout), an easy test given that rollout points at the first eojeols of the page, and did so within correct and within incorrect predictions alike (4.33 against 1.67 and 3.67 against 1.67). With the document in view, the raters judged prediction correctness in agreement with the scoring rule on 88–92% of the items and identified 9–11 of the 15 incorrect predictions. Three limitations bound this reading; each affects the results in a specific way. First, because the raters are not practitioners or document AI researchers, the reliability and the separation between methods hold for non-specialist raters trained on 10 practice items; they do not show that practitioners would agree with each other or with these raters, and the absolute scores are specific to this rater group (the second annotator’s familiarity with the benchmark is addressed by the sensitivity analysis in Supplementary Document S2, which excludes items from doubly annotated documents). Second, no rater applied the not-applicable clause of the semantic consistency dimension, so this dimension was, in effect, scored on every item, based on whether the explanation localized the field context, which is the likely reason for its higher agreement and means that the dimension as defined in Section 3.3.4 was not measured. Eligibility for that dimension had also been defined by whether the document carried several dispatch copies rather than by values of the same type in different roles. Third, there was no condition without explanations, so the correctness-judgment figures describe raters who had the whole document in view and cannot be attributed to the explanations. The pilot study supports the feasibility of the protocol and the discrimination of the rubric between attribution methods, not the usefulness of explanations to a reviewer. Beyond these three, the sample is one model, one seed, and two methods. The full results, the completeness checks, and two sensitivity analyses (excluding items from doubly annotated documents, and items with no extracted value) are presented in Supplementary Document S2. The contribution claimed for the layer remains the design.

6. Conclusions

This study proposed KoDocXAI, a framework for evaluating the explanations of key information extraction models with multimodal input over Korean administrative form documents, together with a pilot benchmark for validating it. The framework evaluates explanations; it does not propose an extraction method.

The validated contribution is the quantitative layer of automatically computable metrics; the qualitative layer, for dimensions that resist automation, is a proposed rubric and protocol. In the quantitative layer, a value-region rationale reference R_k—the region of the extracted value—is derived directly from KIE annotations, quantifying value-region plausibility without dedicated rationale annotation, and faithfulness is measured in decomposed form via modality-wise perturbations of text, layout, and image. The basic unit of attribution is re-aggregated from subwords to morphemes and eojeols, treating the aggregation level in an agglutinative language as a manipulated variable.

The KoDocXAI pilot benchmark v1.0 consists of value-span and bounding-box annotations—the value-region rationale reference—for five fields over 300 permit-and-authorization dispatch documents, with 20% doubly annotated for inter-annotator agreement. The annotation guideline and the annotation tool are released together.

The answers to the three research questions are as follows. RQ1: On the noun-phrase-dominant corpus of this benchmark, the aggregation level (subword, morpheme, eojeol) did not materially change the faithfulness scores or the ranking of methods, and what had appeared to be an aggregation-level effect was an artifact of self-evidence masking (Section 5.1). RQ2: The value-region rationale reference derived from KIE annotations discriminated the value-region plausibility of methods without dedicated rationale annotation: IG far exceeded the random baseline, while attention rollout showed no meaningful advantage over it on either axis under either a first-token or a span-level target; the ranking is unchanged when the reference region is dilated by up to 50% (Section 5.2), and it holds for two further attribution methods, five training seeds and a second backbone without visual features (Section 4.3). RQ3: Modality-wise perturbation separately quantified prediction sensitivity to text, layout, and image—interventions that are not comparable in strength across modalities—revealing the inactivity of the image modality in this setting (which confines the validated scope of the framework to the text and layout channels), a layout pattern consistent with dependence on relative arrangement, and field-level differences extending to the weak context dependence and value-and-position-centered decisions that the issuing authority exhibits in normal documents (Section 5.3 and Section 5.4).

Scope and limitations. Two kinds of statements should be kept apart. The design properties of the framework—separate reporting of value-region plausibility and faithfulness, re-aggregation to Korean units, protection of the ground-truth span, and modality-wise perturbation—do not depend on the corpus. The empirical findings—the ranking of IG over rollout, the inert image channel, the field-wise modality profiles, and the reported numbers—were obtained on one document type (permit-and-authorization dispatch documents, mostly from a single local government), one main model (LayoutXLM-base; five training seeds, with the method ranking reproduced on a second backbone without visual features under one seed), and a corpus with specific structural properties (several dispatch copies on 23.0% of pages, seal-overlapped values, and six instances truncated at 512 tokens); they apply to Korean administrative form documents of this type and to LayoutXLM-based KIE, and the numbers are not universal reference values. The perturbation findings are conditional on the operational definitions adopted, the qualitative layer has been applied only in a pilot study and remains unvalidated, and full reconstruction of the benchmark is domestically restricted (Section 5.5). Two consequences stated in Section 5.2 and Section 5.5 apply to these conclusions: the plausibility values are conditional on the value-region definition of the reference and should not be compared with scores obtained against human rationales, and the pilot of the qualitative layer supports feasibility and discrimination among methods for non-specialist raters, not the usefulness of explanations to reviewers.

We also reported methodological observations from the construction process. When a field definition fails to cover a case, two annotators can err in the same direction, and inter-annotator agreement then rises rather than falls; high agreement therefore does not guarantee the validity of annotation rules, and definition-completeness checks are needed separately from agreement computation (Section 5.4). The “one draft, multiple dispatches” practice of Korean official documents places multiple dispatch copies on a single document image, creating a multi-instance problem in the definition of evidence ground truth (Section 5.2). To our knowledge, neither observation has been reported in existing form-document benchmarks, and both bear directly on the construction of Korean document resources.

Extension directions. Five extensions are planned for the next release of the KoDocXAI benchmark: document groups with different degrees of structural regularity, to delineate where evidence ground truth can be derived automatically (RQ2) and, in text-richer corpora, to broaden the verification scope of RQ1; scale, institutions, and vintage, to verify replication on contemporary electronic documents—where fine-grained rankings among methods are the goal, increasing the number of seeds takes priority over the number of documents (Supplementary Document S3); per-instance annotation of multiple dispatch copies with multi-instance matching metrics, to score which document the model grounded its prediction in; a controlled cross-lingual contrast applying the same model to isomorphic forms in two languages; and evaluation of further document understanding architectures, in particular a known-groups design with a model in which visual features demonstrably matter and documents that contain images, to separate properties of the framework from properties of the backbones evaluated here and of this corpus. Validation of the value-region reference against human rationales on a subset, and application of the qualitative-layer protocol with practitioner raters and a condition without explanations, are likewise left to follow-up work, as are the resolution of routing notations to the entities they denote and the restoration of damaged strings, which would address the gap between the surface extraction and semantic answers quantified in Section 5.5.