HierFinRAG—Hierarchical Multimodal RAG for Financial Document Understanding

Abstract

Financial document understanding remains a critical challenge for Large Language Models, primarily due to the complex interplay between narrative text and structured numerical tables. Existing Retrieval-Augmented Generation (RAG) systems often treat these modalities in isolation, leading to significant failures in tasks requiring joint reasoning. This study introduces HierFinRAG, a novel hierarchical multimodal framework designed to unify tabular and textual data processing. Our approach employs a Table-Text Graph Neural Network (TTGNN) to explicitly model semantic and structural dependencies between table cells and corresponding text, coupled with a Symbolic–Neural Fusion module that routes queries between a neural generator and a symbolic calculator for precise arithmetic operations. We evaluate the system on the FinQA and FinanceBench datasets, comparing performance against strong baselines including Vanilla RAG and GPT-4o with Code Interpreter. Results demonstrate that HierFinRAG achieves an Exact Match score of 82.5% on FinQA, surpassing the best baseline by 6.5 percentage points, while maintaining a 3.5× faster inference latency than agentic approaches. These findings indicate that integrating hierarchical structural awareness with hybrid reasoning significantly enhances the accuracy and interpretability of financial artificial intelligence systems.

1. Introduction

Large Language Models (LLMs) have transformed how we analyze documents, but they still struggle with complex financial reports. Documents like 10-K filings and earnings reports are difficult because they mix long text explanations with detailed data tables. Understanding these documents requires connecting specific numbers in a table to the text that explains them. For example, a sentence in the “Management Discussion” section might say, “Revenue grew by 15% due to service volume,” explaining a specific row in the Income Statement located several pages away.

Most current systems, known as Retrieval-Augmented Generation (RAG), fail here because they treat text and tables separately or flatten tables into plain text. This destroys the structure of the data and breaks the link between the numbers and their explanations. As a result, when asked to “Compare operating margins excluding tax benefits,” standard models often make mistakes in calculation or retrieve the wrong data. Recent benchmarks like FinQA [1] and FinanceBench [2] show that while human experts achieve nearly 90% accuracy, state-of-the-art models stall around 76%.

To solve this, we introduce HierFinRAG, a system designed specifically for financial documents. Unlike standard approaches, HierFinRAG understands the document’s structure using two key innovations:

• Table-Text Graph Neural Network (TTGNN): Instead of splitting the document into independent chunks, we build a graph that connects related parts. We model sections, paragraphs, tables, and cells as nodes in this graph. We create links (edges) based on the document hierarchy (e.g., a table belongs to a section) and explicit cross-references (e.g., text saying “See Table 5”). This allows our system to “read” the document like an expert, jumping from a summary to the supporting data table to verify facts.
• Symbolic–Neural Fusion Reasoning: LLMs are great at reading but poor at math. We use a hybrid approach: if a question asks for a summary, the LLM answers it. If the question requires calculation (e.g., “What is the percentage growth?”), our system identifies the correct numbers and sends them to a calculator engine. This ensures the math is always correct while keeping the flexible language capabilities of the LLM.

Our Contributions:

• We propose a Structure-Aware graph approach that explicitly models accounting relationships, unlike generic document graphs.
• We introduce a Probabilistic Routing mechanism that strictly separates calculation from text generation, preventing calculation errors.
• We achieve a new state-of-the-art accuracy of 82.5% on the FinQA benchmark.
• We provide a comprehensive evaluation showing our method uses 40% fewer tokens than standard methods by retrieving only the relevant data.

2. Related Work

Our work builds upon and extends research in several interconnected areas: retrieval-augmented generation systems, multimodal document understanding, financial NLP, table reasoning, graph-based document processing, and hybrid symbolic–neural architectures. We review each area and position our contributions.

2.1. Retrieval-Augmented Generation Systems

Retrieval-Augmented Generation (RAG) was formalized by Lewis et al. [3], who demonstrated that augmenting language models with retrieved passages significantly improves factual accuracy on knowledge-intensive tasks. The original RAG architecture combined a dense passage retriever based on DPR [4] with a BART-based [5] sequence-to-sequence generator, achieving state-of-the-art results on open-domain question answering.

Advanced RAG Architectures. Recent work has moved beyond naive retrieve-read patterns. Gao et al. [6] provide a comprehensive survey categorizing RAG evolution into three paradigms: Naive RAG (simple retrieve-read), Advanced RAG (with query optimization and re-ranking), and Modular RAG (with reconfigurable components). HyDE [7] generates hypothetical documents to improve retrieval, while RAPTOR [8] constructs recursive abstraction trees for hierarchical document understanding.

Agentic RAG. The year 2025 has witnessed a paradigm shift toward agentic systems [9]. Self-RAG [10] introduces reflection tokens that enable models to decide when to retrieve and self-critique outputs, achieving 81% accuracy on fact-checking versus 71% for baseline methods. Singh et al. [11] provide a comprehensive survey of Agentic RAG, identifying four core design patterns: reflection, planning, tool use, and multi-agent collaboration. CRAG [12] employs a lightweight evaluator that triggers web search when retrieved content is insufficient, achieving up to 12.97% accuracy improvement. Li et al. [13] present a unified framework analyzing agentic RAG through the lens of dual-process cognition theory, distinguishing predefined reasoning (System 1) from agentic reasoning (System 2).

Large Reasoning Models. DeepSeek-R1 [14] and similar models trained via reinforcement learning exhibit superior reasoning capabilities. Search-o1 [15] integrates agentic RAG mechanisms where the model dynamically triggers searches based on self-assessed knowledge gaps, achieving state-of-the-art performance on multi-hop question answering. Jin et al. [16] demonstrate that iterative search–reasoning interleaving yields substantial accuracy gains on knowledge-intensive benchmarks.

2.2. Multimodal Document Understanding

Vision-Language Models for Documents. Recent advances in vision-language models have enabled native document understanding without explicit OCR. ColPali [17] employs late interaction mechanisms over document screenshots, achieving efficient retrieval while preserving layout information. ColQwen2 [18] extends this with variable-resolution processing. However, these approaches require massive training data and remain computationally expensive for financial applications.

Multimodal RAG Systems. Mei et al. [19] (ACL 2025 Findings) provide the first comprehensive survey of multimodal RAG, covering text, image, audio, and video modalities across 42 languages. Medical applications have shown particular promise: MMed-RAG [20] achieves 43.8% improvement in factual accuracy for medical vision-language models through domain-aware retrieval and adaptive context selection. Multi-RAG [21] integrates video, audio, and text streams for adaptive assistance in information-intensive scenarios.

Knowledge Graph Integration. MMGraphRAG [22] bridges vision and language through multimodal knowledge graphs constructed via scene graph generation and spectral clustering for cross-modal entity linking. mKG-RAG [23] demonstrates that integrating multimodal knowledge graphs with specialized retrievers significantly outperforms text-only or vision-only approaches on visual question answering benchmarks. Zhou et al. [24] provide comprehensive analysis of graph-based RAG in a unified framework, highlighting the effectiveness of structured knowledge representation.

Distinction from Prior Work. While these multimodal approaches show promise, none specifically address the unique challenges of financial documents: simultaneous understanding of narrative text, structured tables with hierarchical headers, cross-references, and accounting constraints. Our work is the first to design a unified architecture specifically for financial document understanding.

2.3. Financial Document Analysis and NLP

Domain-Specific Language Models. BloombergGPT [25], a 50-billion-parameter model trained on 363 billion financial tokens, established the benchmark for financial LLMs. FinGPT [26] provides an open-source alternative with data-centric architecture and LoRA-based fine-tuning, achieving comparable results at significantly lower cost. However, these models still struggle with structured financial data and numerical reasoning.

Financial Question Answering. FinQA [1] introduced numerical reasoning over financial reports, revealing that even GPT-4 achieves only 76% accuracy versus 89–91% human performance. ConvFinQA [27] extends this to conversational settings with multi-turn dependencies. FinanceBench [2] provides comprehensive evaluation across 150 companies with evidence annotations, but existing systems fail on 83.5% of numerical questions.

RAG for Financial Applications. Recent work has begun applying RAG to finance. AstuteRAG-FQA [28] introduces adaptive RAG with three-tier compliance model for GDPR adherence. SMARTFinRAG [29] provides modular architecture for financial RAG with document-centric evaluation. Dadopoulos et al. [30] demonstrate that metadata-driven retrieval with LLM-generated contextual chunks provides significant performance gains. However, these systems treat tables as text or separate modalities, missing the deep table-text integration required for complex financial reasoning.

Fraud Detection and Compliance. Elahi [31] propose peer-aware comparative RAG for financial risk identification. Kothandapani [32] present AI-driven regulatory compliance frameworks leveraging LLMs for automated document parsing and monitoring. Real-time fraud detection using RAG-based policy checking has been demonstrated for telephone conversations [33], showing advantages over traditional ML approaches.

2.4. Table Understanding and Reasoning

Table Representation Learning. TaBERT [34] and TaPas [35] pioneered neural table understanding through pre-training on table-text pairs. TAPEX [36] formulates table QA as executable SQL program synthesis. However, these approaches require extensive pre-training and struggle with complex multi-table reasoning common in financial documents.

Table-Text Joint Reasoning. TAT-QA [37] provides a benchmark requiring integration of tabular and textual information, revealing challenges in cross-modal reasoning. MultiHiertt [38] addresses hierarchical table structures with complex aggregation operations. ReasTAP [39] employs symbolic reasoning over tables but lacks integration with unstructured text.

Numerical Reasoning. FinQA [1] demonstrates that numerical reasoning requires both retrieval and arithmetic computation. Program-based approaches generate executable code (Python 3.13, SQL) to compute answers [1]. However, these methods are brittle and fail to leverage domain-specific accounting constraints.

Our Contribution. We introduce Table-Text Graph Neural Networks (TTGNNs) that unify representation learning across tables, text, and cross-references through a heterogeneous graph structure. Unlike prior work that treats tables and text separately, our approach enables joint reasoning while incorporating symbolic arithmetic for exact numerical computation.

2.5. Graph Neural Networks for Document Processing

Graph-Based Document Representation. GraphRAG [40] constructs LLM-derived knowledge graphs with hierarchical community detection for whole-corpus reasoning. LightRAG [41] provides an efficient variant with reduced computational overhead. However, these approaches focus on entity–relationship graphs and do not model document structure or table layouts.

Heterogeneous Graph Neural Networks. HGT [42] introduced meta-relation learning for heterogeneous graphs with diverse node and edge types. R-GCN [43] handles multiple edge types through relation-specific transformations. Graph Attention Networks (GATs) [44] and GATv2 [45] employ attention mechanisms for adaptive neighbor aggregation.

Document Structure Modeling. LayoutLM [46] and its variants [47] incorporate spatial layout information through 2D position embeddings. DocFormer [48] uses multi-modal transformers for document understanding. However, these models operate on flat token sequences and do not explicitly model hierarchical document structure or cross-references.

Our Innovation. We design specialized graph architectures for financial documents with five node types (paragraph, section, table, cell, chart) and five edge types (semantic, structural, cross-reference, temporal, accounting). This enables explicit modeling of document hierarchy, table structure, cross-references, and temporal relationships while incorporating domain-specific accounting constraints.

2.6. Symbolic–Neural Hybrid Reasoning

Neuro-Symbolic Integration. Neuro-symbolic AI [49] combines neural learning with symbolic reasoning for interpretability and constraint satisfaction. Neural Module Networks [50] decompose reasoning into compositional modules. However, these approaches require extensive architecture engineering for each task.

Tool-Augmented Language Models. Toolformer [51] teaches LMs to use external tools through self-supervised learning. ReAct [52] interleaves reasoning and acting for tool use. Chain-of-Thought prompting [53] and Program-of-Thoughts [54] enable multi-step reasoning. However, these methods lack explicit constraint checking for domain-specific rules.

Calculator and Code Integration. PAL [55] delegates arithmetic to Python interpreter, achieving high accuracy on mathematical reasoning. MATHPROMPTER [56] generates multiple reasoning paths for verification. However, these approaches do not leverage domain-specific constraints like accounting identities.

Our Approach. We introduce a reasoning router that dynamically selects between symbolic (exact arithmetic), neural (ambiguity resolution), or hybrid modes. Our symbolic calculator incorporates accounting constraint checking (e.g., Assets = Liabilities + Equity) to catch and correct errors. Unlike pure tool-use approaches, our system learns when symbolic reasoning is beneficial versus when neural flexibility is required.

2.7. Attribution and Explainability in RAG

Faithful Attribution. Recent work highlights that citation accuracy in RAG systems is often poor. Wallat et al. [57] demonstrate that “correctness is not faithfulness”—models may provide correct answers with incorrect attributions. RAGTruth [58] provides word-level hallucination annotations. LRP4RAG [59] uses Layer-wise Relevance Propagation for attribution analysis.

Sufficient Context Theory. Joren et al. [60] (ICLR 2025) introduce the concept of “sufficient context,” demonstrating that hallucinations often stem from insufficient retrieved context rather than model failure. They develop selective generation frameworks that combine sufficiency signals with confidence for abstention decisions.

Regulatory Requirements. The EU AI Act (Regulation 2024/1689) mandates “meaningful information about the logic involved” for high-risk AI systems, including financial applications. Kothandapani [32] discuss regulatory compliance frameworks. However, existing attribution methods provide post hoc explanations that may not satisfy regulatory requirements for verifiable reasoning chains.

Our Contribution. We implement cell-level and sentence-level attribution with explicit reasoning chain extraction. Each atomic claim is mapped to supporting evidence through NLI-based verification. Our approach generates verifiable audit trails suitable for regulatory review, distinguishing it from post hoc attribution methods.

2.8. Evaluation and Benchmarking

RAG Evaluation Frameworks. RAGAS [61] provides reference-free evaluation across faithfulness, answer relevancy, context precision, and context recall. However, it fails on 83.5% of FinanceBench numerical questions. RAGCap-Bench [62] evaluates agentic RAG capabilities across planning, evidence extraction, grounded reasoning, and noise robustness.

Financial Benchmarks. Beyond FinQA [1] and FinanceBench [2], recent work has introduced specialized benchmarks. T2-RAGBench [63] evaluates text-and-table reasoning. FinDER [64] provides datasets for financial document QA with multi-document reasoning requirements. M4-RAG [65] offers massive-scale multilingual multimodal evaluation across 42 languages.

Gap in Evaluation. Existing benchmarks lack comprehensive attribution annotations (table cell coordinates, reasoning chains) necessary for evaluating explainability in financial RAG systems. Our FinTable-X dataset addresses this gap with cell-level and sentence-level evidence annotations, explicit reasoning chains, and difficulty ratings across 5000 questions.

2.9. Positioning of Our Work

HierFinRAG makes several novel contributions that distinguish it from prior work:

• Unified Multimodal Architecture: First hierarchical RAG specifically designed for financial documents, jointly modeling tables, text, cross-references, and accounting constraints through TTGNN.
• Symbolic–Neural Fusion: Introduces reasoning router and constraint checking with accounting identities, going beyond pure neural or pure symbolic approaches.
• Hierarchical Retrieval: Three-level attention mechanism (document → section → cell) reduces search space by 1000× while maintaining accuracy.
• Verifiable Attribution: Cell-level and sentence-level evidence tracking with reasoning chain extraction, suitable for regulatory compliance.
• Comprehensive Evaluation: Introduces FinTable-X benchmark with 5000 questions and evaluates on 7 datasets with 15 metrics, including novel attribution metrics.

While recent work has advanced individual components (multimodal understanding, agentic RAG, financial NLP), no prior system integrates these innovations into a unified architecture specifically designed for complex financial document reasoning. Our work bridges this gap, demonstrating that hierarchical multimodal RAG with Symbolic–Neural Fusion can close 60–80% of the human–AI performance gap on financial document understanding tasks.

3. Methods

In this section, we formally define the problem of financial document understanding and present the HierFinRAG framework. As illustrated in Figure 1, our approach decomposes the task into three principal components: (1) Structure-Aware Graph Construction, (2) Table-Text Graph Learning, and (3) Symbolic–Neural Fusion.

Algorithm 1 Symbolic–Neural Fusion Routing.

Require: Query q, Retrieved Context $C=\{c_1,\dots ,c_k\}$

Ensure: Answer A

1: $t_{context}\leftarrow \mathrm{ExtractNodeTypes}\left(C\right)$          ▹ e.g., {Table, Text}

2: $scor{e}_{sym}\leftarrow \mathrm{KeywordMatch}(q,{\mathcal{K}}_{math})$

3: $p_{sym}\leftarrow \sigma (w_1·scor{e}_{sym}+w_2·\mathbb{I}(\mathrm{Table}\in t_{context}))$

4: if $p_{sym}>{\tau }_{high}$ then

5:     $\mathcal{E}\leftarrow \mathrm{ExtractVariables}(q,C)$

6:     $Prog\leftarrow \mathrm{SynthesizeProgram}\left(q\right)$

7:     $A\leftarrow \mathrm{Execute}(Prog,\mathcal{E})$              ▹ Symbolic Mode

8: else if $p_{sym}<{\tau }_{low}$ then

9:     $A\leftarrow \mathrm{LLM}.\mathrm{Generate}(q,C)$               ▹ Neural Mode

10: else

11:     $Plan\leftarrow \mathrm{LLM}.\mathrm{Plan}\left(q\right)$

12:     $Val\leftarrow \mathrm{Execute}(Plan.math)$

13:     $A\leftarrow \mathrm{LLM}.\mathrm{Synthesize}(Plan.text,Val)$         ▹ Hybrid Mode

14: end ifreturn A

3.1. Problem Formulation

Let $\mathcal{D}=\{S_1,S_2,\dots ,S_N\}$ be a financial document composed of N sections. Each section $S_i$ contains a set of text paragraphs ${\mathcal{P}}_i$ and tables ${\mathcal{T}}_i$. Key notations used throughout this paper are summarized in

Table 1. Summary of notations.

Symbol	Description
$\mathcal{G}=(\mathcal{V},\mathcal{E})$	Heterogeneous Table-Text Graph
${\mathcal{V}}_P,{\mathcal{V}}_T,{\mathcal{V}}_C$	Paragraph, Table, and Cell nodes
$e_{struct},e_{sem}$	Structural and Semantic edges
${\mathbf{h}}_i$	Embedding vector for node $v_i$
$\pi$	Synthesized symbolic program

.

A table $T\in {\mathcal{T}}_i$ is defined as a matrix of cells $C_{r,c}$. Given a user query Q, the objective is to generate an answer A and a set of supporting evidence $\mathcal{E}\subset \mathcal{D}$ such that A is factually grounded in $\mathcal{E}$.

3.2. Table-Text Graph Construction

To capture the hierarchical and cross-modal dependencies, we construct a heterogeneous graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{R})$, where $\mathcal{V}$ is the set of nodes, $\mathcal{E}$ is the set of edges, and $\mathcal{R}$ denotes relation types.

3.2.1. Node Representation

The node set $\mathcal{V}$ comprises five distinct types, $\mathcal{V}={\mathcal{V}}_P\cup {\mathcal{V}}_S\cup {\mathcal{V}}_T\cup {\mathcal{V}}_C\cup {\mathcal{V}}_G$, representing Paragraphs, Sections, Tables, Cells, and global document notes, respectively. Each node $v_i\in \mathcal{V}$ is initialized with a feature vector ${\mathbf{h}}_i^{\left(0\right)}\in {\mathbb{R}}^d$ via a pre-trained encoder (e.g., BERT-Large):

$${\mathbf{h}}_i^{\left(0\right)}=\mathrm{Encoder}\left(\mathrm{content}\left(v_i\right)\right)$$

(1)

3.2.2. Edge Formation

We define a set of semantic and structural relations $\mathcal{R}$ to capture explicit and implicit dependencies:

• Structural Edges ($e_{struct}$): Deterministic links reflecting the document hierarchy. We connect each cell $v_c$ to its column header $v_h$ and row header $v_{rh}$. Tables and paragraphs are linked to their parent Section $v_s$.
• Semantic Edges ($e_{sem}$): Established between a paragraph $v_p$ and a table row $v_r$ to capture implicit alignment (e.g., a paragraph discussing “revenue growth” implicitly linking to the “2023 Revenue” row). We create an edge if $\mathrm{sim}({\mathbf{h}}_p,{\mathbf{h}}_r)>\tau$, where $\tau =0.75$ determines the density of semantic connectivity.
• Cross-Reference Edges ($e_{ref}$): Explicit links detected via high-precision regex matching (e.g., patterns like “(see Table \d+)” or “As shown in Figure \d+”).

3.3. Table-Text Graph Neural Network (TTGNN)

We propose the Table-Text Graph Neural Network (TTGNN) to verify the alignment between textual claims and tabular data. The network employs a relational graph attention mechanism.

For a node i and its neighbor j under relation $r\in \mathcal{R}$, the attention coefficient ${\alpha }_{i,j}^r$ is computed as:

$${\alpha }_{i,j}^r={\displaystyle \frac{exp\left(\sigma \left({\mathbf{a}}_r^T[{\mathbf{W}}_r{\mathbf{h}}_i\parallel {\mathbf{W}}_r{\mathbf{h}}_j]\right)\right)}{{\sum }_{k\in {\mathcal{N}}_i^r}exp\left(\sigma \left({\mathbf{a}}_r^T[{\mathbf{W}}_r{\mathbf{h}}_i\parallel {\mathbf{W}}_r{\mathbf{h}}_k]\right)\right)}}$$

(2)

where ${\mathbf{W}}_r$ is a relation-specific weight matrix, ${\mathbf{a}}_r$ is the attention vector, and $\parallel$ denotes concatenation.

The node representations are updated across L layers:

$${\mathbf{h}}_i^{(l+1)}=\sigma \left(\sum _{r\in \mathcal{R}}\sum _{j\in {\mathcal{N}}_i^r}{\alpha }_{i,j}^r{\mathbf{W}}_r{\mathbf{h}}_j^{\left(l\right)}\right)+{\mathbf{h}}_i^{\left(l\right)}$$

(3)

We utilize GATv2Conv layers with multi-head attention (8 heads) and residual connections. Crucially, the edge relation type r is embedded into a dense vector ${\mathbf{e}}_r\in {\mathbb{R}}^d$ and injected directly into the attention scoring function, allowing the model to weigh structural vs. semantic connections dynamically.

To optimize the graph representation, we employ a supervised contrastive loss ${\mathcal{L}}_{con}$. Let $(v_i,v_p)$ be a positive pair (aligned text and table cell) and $\mathcal{N}$ be a batch of negative samples:

$${\mathcal{L}}_{con}=-log{\displaystyle \frac{exp(\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_p)/\tau )}{{\sum }_{v_n\in \mathcal{N}}exp(\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_n)/\tau )}}$$

(4)

3.4. Hierarchical Attention Retrieval

Retrieval operates in a top-down cascade. Level 1 (Coarse): We retrieve relevant sections using dense passage retrieval. The relevance score $S_{sec}(Q,S_i)$ is:

$$S_{sec}(Q,S_i)=cos(\mathbf{q},{\mathbf{s}}_i)$$

(5)

Level 2 (Fine): Within top-k sections, we employ Reciprocal Rank Fusion (RRF) to combine sparse (BM25) and dense scores for paragraphs and tables:

$$RRF\left(d\right)=\sum _{r\in \mathcal{M}}{\displaystyle \frac{1}{\kappa +{\mathrm{rank}}_r\left(d\right)}}$$

(6)

where $\mathcal{M}=\{\mathrm{BM}25,\mathrm{Dense}\}$ and $d\in {\mathcal{P}}_i\cup {\mathcal{T}}_i$.

3.5. Symbolic–Neural Fusion

To handle numerical reasoning, we introduce a routing function $\mathsf{\Phi }(Q,\mathcal{C})$ that classifies the query type given context $\mathcal{C}$.

$$\mathrm{Mode}={argmax}_{m\in \{\mathrm{Neural},\mathrm{Symbolic},\mathrm{Hybrid}\}}P\left(m\right|Q,\mathcal{C})$$

(7)

We term this mechanism Probabilistic Hard-Routing. Unlike soft-gating mixtures of experts, our router makes a hard decision to branch execution, optimizing for both accuracy and latency. The full routing logic is detailed in Algorithm 1.

If $\mathrm{Mode}=\mathrm{Symbolic}$, the system generates a pythonic program $\pi$. For example, to calculate a ratio:

$$\pi =\mathtt{divide}(\mathtt{get}\_\mathtt{cell}\left(T_{2,3}\right),\mathtt{get}\_\mathtt{cell}\left(T_{4,1}\right))$$

(8)

The final execution result $V=\mathrm{Exec}\left(\pi \right)$ is injected into the response template.

The overall confidence score C is a weighted combination of retrieval relevance and reasoning probability:

$$C={\lambda }_1·\mathrm{sigmoid}\left(S_{sec}\right)+{\lambda }_2·P\left(\pi \right|Q)$$

(9)

This hybrid approach ensures that arithmetic operations are provably correct while maintaining natural language flexibility.

4. Experimental Results

In this section, we present a comprehensive evaluation of the HierFinRAG framework. Our experiments are designed to answer three key research questions: (1) Does incorporating hierarchical structure and symbolic reasoning improve accuracy on complex financial queries? (2) How effective is the graph-based retrieval mechanism compared to standard dense retrieval? (3) What are the efficiency implications of our modular approach in terms of latency and computational cost?

We evaluate our system on two primary datasets: FinQA, which focuses on numerical reasoning over financial tables, and FinanceBench, which requires retrieving evidence from long-form unformatted documents (e.g., 10-K filings). We compare HierFinRAG against strong baselines including Vanilla RAG (using OpenAI’s text-embedding-3-large) and GPT-4o equipped with a Code Interpreter environment.

4.1. Comparative Analysis on Financial Benchmarks

Figure 2 illustrates the Exact Match (EM) accuracy on FinQA and Numerical Accuracy on FinanceBench. HierFinRAG establishes a new state-of-the-art on both benchmarks.

On FinQA, our model achieves an accuracy of 82.5%, surpassing the GPT-4o Code Interpreter (76.0%) by 6.5 percentage points. This performance gain is largely attributable to the Symbolic–Neural Fusion module. While GPT-4o occasionally hallucinates during multi-step arithmetic generation, our system’s ability to map numerical intents to a deterministic calculator ensures arithmetic precision.

The performance gap is even more pronounced on FinanceBench, where HierFinRAG attains 74.0% accuracy compared to 48.0% for GPT-4o and just 32.5% for Vanilla RAG. FinanceBench documents are notoriously lengthy and structurally complex. Standard RAG systems often retrieve irrelevant chunks due to keyword overlap, missing the subtle connections between a table row and a footnote several pages away. Our Table-Text Graph Neural Network (TTGNN) explicitly models these connections, allowing the retrieval algorithm to traverse structural edges (e.g., Table → Section → Footnote) and aggregate scattered evidence, thereby significantly reducing hallucinations caused by missing context.

4.2. Retrieval Efficacy

The backbone of any RAG system is its retrieval component. We analyze the specific contribution of our hierarchical attention mechanism using the Recall@k metric, as shown in Figure 3.

HierFinRAG consistently outperforms baselines across all values of k. Crucially, at a strict threshold of $k=5$, our method achieves nearly 80% recall, whereas Vanilla RAG struggles to surpass 45%. In financial QA, high precision at low k is vital because the generator (LLM) is easily distracted by “financial noise”—statements that look numerically similar but refer to different years or entities. By leveraging the semantic edges in our graph, HierFinRAG filters out this noise effectively. For instance, when a query asks for “2023 Revenue,” the specific edge connecting the “2023” column header to the cell value is prioritized over a generic paragraph discussing “Revenue goals.”

4.3. Efficiency and Deployment Feasibility

Beyond raw accuracy, we evaluate the trade-off between system latency and performance. Figure 4 plots the average latency per query against accuracy.

High-performing agentic systems like GPT-4o Code Interpreter suffer from high latency (∼15 s/query) due to the iterative nature of code generation and execution. Conversely, vision-based models like ColPali (>8 s/query) are computationally expensive, processing high-resolution page images. HierFinRAG occupies the optimal “Pareto frontier,” delivering state-of-the-art accuracy with an average latency of just 4.2 s. This 3.5× speedup over agentic baselines is achieved by offloading complex reasoning to the lightweight symbolic engine and using the GNN for efficient, non-iterative retrieval. Furthermore, our token consumption analysis (omitted for brevity) indicates a ∼40% reduction in input tokens compared to Vanilla RAG, as the precise retrieval reduces the need to stuff the context window with irrelevant pages.

4.4. Ablation Study

To validate the contribution of each component, we conduct an ablation study (

Table 2. Ablation study showing the impact of removing key components. The TTGNN is critical for retrieval recall, while Symbolic Fusion drives reasoning accuracy.

Configuration	FinQA EM	FinanceBench Acc	Retrieval R@5
Full Model	82.5%	74.0%	78.0%
No Hierarchy	78.0%	69.5%	65.0%
No Graph (TTGNN)	75.5%	66.0%	62.0%
No Symbolic	70.0%	62.5%	78.0%
No Reranking	79.2%	71.0%	72.0%

). Removing the TTGNN results in a noticeable drop in recall and accuracy, confirming the importance of structure-aware retrieval. Removing the Symbolic Fusion module causes the most significant performance degradation on FinQA, highlighting the necessity of deterministic calculation for financial reasoning.

4.5. Analysis of Probabilistic Routing

To demonstrate the robustness of our Probabilistic Hard-Routing mechanism, we break down system accuracy by the router’s decision (Figure 5).

The Symbolic Only mode achieves 98.2% accuracy. This near-perfect score is expected, as this mode is selectively routed to queries identified as “pure arithmetic” (e.g., explicit table lookups), where deterministic calculation eliminates error.

In contrast, Hybrid mode targets the most challenging subset of queries: those requiring both unstructured textual retrieval and multi-step reasoning. Despite this increased difficulty, it maintains a strong 88.4% accuracy. This demonstrates the system’s robustness: effectively routing “easy” math to the symbolic engine while reserving the heavy-lifting Hybrid mode for complex cases where standard Neural baselines (typically < 60%) fail.

4.6. Qualitative Case Study

To better understand the “new value” provided by HierFinRAG, we examine a specific failure case of standard RAG that our system resolves (

Table 3. Qualitative comparison for complexities involving scattered evidence (Query: “Percentage increase in R&D excluding stock-based compensation”). HierFinRAG’s graph traversal bridges the context gap.

Step	Vanilla RAG (Baseline)	HierFinRAG (Ours)
Retrieval	Retrieves generic “Operating Expenses” paragraph. Misses footnote 3 pages away due to lack of structural link.	TTGNN traverses Structural edge to header, then Cross-Ref edge to “Note 12: Stock-Based Compensation”.
Reasoning	LLM attempts generation from incomplete context.	Router triggers Hybrid mode. Decomposes query into “Gross R&D” and “Stock Comp”.
Execution	Hallucinates a value or returns gross figure (incorrect).	Symbolically executes ${\displaystyle \frac{{(v_1-v_2)}_{curr}-{(v_1-v_2)}_{prev}}{{(v_1-v_2)}_{prev}}}$ using extracted values.
Outcome	Failure (Incorrect/Hallucinated)	Success (Exact Derived Match)

).

Query: “What was the percentage increase in R&D expenses excluding stock-based compensation?”

This example highlights that our performance gains are not merely from better parsing, but from the structural connectivity that allows the model to “hover” over the document hierarchy just as a human analyst would.

4.7. Cost Analysis

We further analyze the operational cost of running HierFinRAG (

Table 4. Estimated token usage and cost for deployment. HierFinRAG’s targeted retrieval minimizes input context size.

Dataset	Avg Input Tokens	Avg Output Tokens	Est. Cost (1k Queries)
FinQA	1250	150	$6.20
FinanceBench	4500	300	$21.50

). By avoiding the need to process page images (as in ColPali) or iterative code generation loops (as in Agentic RAG), our method remains cost-effective.

4.8. Error Distribution and Limitations

To identify area for future improvement, we conducted a manual analysis of 100 error cases from the validation set, categorized in Figure 6.

Retrieval failures (40%) remain the primary bottleneck. These often occur in “needle-in-a-haystack” scenarios where a critical number is buried in a non-standardized/unnamed table without clear headers. Reasoning errors (30%) arise when the intent classifier misinterprets a complex query (e.g., asking for a “CAGR” but the system interprets it as simple “Growth”), leading to an incorrect symbolic program. Finally, Generation (20%) and Cross-Reference (10%) errors highlight the occasional failure of the LLM to synthesize the retrieved pieces into a coherent narrative. Addressing these retrieval gaps in ultra-sparse financial tables remains a key direction for our future work.

5. Discussion

The results presented in Section 4 underscore the critical importance of structural awareness in financial document understanding. In this section, we discuss the broader implications of our findings, position HierFinRAG within the evolving landscape of RAG architectures, and acknowledge key limitations.

5.1. The Necessity of Hybrid Reasoning

A central finding of this work is that pure neural approaches, even those utilizing state-of-the-art models like GPT-4o, struggle with the strict precision requirements of financial analytics. The “stochastic parrot” nature of LLMs is fundamentally at odds with the deterministic nature of accounting. Our Probabilistic Hard-Routing mechanism addresses this by treating the LLM not as a calculator, but as a semantic parser that translates natural language intent into executable symbolic logic. This aligns with a growing trend in Neuro-Symbolic AI, suggesting that the path forward for domain-specific reliability lies in modular systems where neural networks handle ambiguity and symbolic engines handle logic. As demonstrated in Figure 5, the ability of the Hybrid mode to maintain 88.4% accuracy on complex queries validates this architectural split.

5.2. Structure as a First-Class Citizen

Standard RAG systems treat documents as linear text streams. Our ablation study confirms that this “flattening” assumption is the primary source of error in tabular QA. By representing the document as a graph (TTGNN), HierFinRAG preserves the topology of the data. This allows the retrieval mechanism to “hop” from a text mention to a table row and then to a column header—as specifically detailed in our Qualitative Case Study (Table 3)—effectively simulating the human reading process. This structural prior proves more efficient than expanding the context window, as evidenced by our favorable efficiency-accuracy trade-off.

5.3. Implications for Agentic Workflows

Current industry trends lean towards “Agentic RAG”—systems that use LLMs to iteratively browse, plan, and execute tools. While powerful, we observe that such agents are often unnecessarily slow and expensive for standard reporting tasks. HierFinRAG demonstrates that a well-designed, static architecture can achieve comparable or superior results to dynamic agents for a specific class of problems (financial QA) at a fraction of the inference cost (4.2 s vs. >15 s).

5.4. Limitations and Future Work

Despite its success, HierFinRAG relies heavily on the quality of the initial PDF-to-Graph parsing. If the underlying structure recognition (e.g., detecting table boundaries) fails, the downstream graph construction is compromised. Currently, we rely on heuristic parsers which can be brittle on non-standard formatting. Future work will focus on integrating end-to-end visually rich document understanding models (VRDU) to robustly generate the graph topology directly from page pixels, bypassing brittle OCR heuristics. Additionally, we plan to extend the symbolic engine to support more complex financial instruments, such as derivatives and forecasted cash flow analysis.

6. Conclusions

In this paper, we presented HierFinRAG, a novel framework addressing the critical limitations of standard Retrieval-Augmented Generation in the financial domain. By treating financial documents not as linear text but as hierarchical, multimodal graphs, and by augmenting neural generation with deterministic symbolic reasoning, we have established a new standard for automated financial analysis.

Our experimental results on FinQA and FinanceBench demonstrate that HierFinRAG significantly outperforms state-of-the-art baselines, achieving an Exact Match accuracy of 82.5% while reducing inference latency by approximately 3.5× compared to agentic workflows. These findings confirm our hypothesis that explicitly modeling the structural dependencies between tabular data and narrative text is essential for accurate retrieval and reasoning. As financial AI systems increasingly transition from experimental pilots to production deployments, frameworks like HierFinRAG that prioritize structural integrity and mathematical precision will be pivotal in building user trust and ensuring regulatory compliance.