Recall Mechanism and Multi-Head Attention for Numerical Reasoning

Abstract

Numerical reasoning is a challenging question-answering task in artificial intelligence (AI) that requires both reading comprehension and numerical computation capabilities. Although recent approaches have made significant progress in reasoning, two critical issues remain: (1) information tends to be gradually lost as the network deepens due to the complexity of deep learning models, and (2) the performance of multi-step reasoning is suboptimal, leaving room for improvement in accuracy. To address these issues, we propose a model with a recall mechanism and multi-head attention for numerical reasoning (RMMANR). The recall mechanism prevents the embedding information of questions and passages from being forgotten as the model deepens, while the multi-head attention mechanism analyzes possible solutions for numerical reasoning. RMMANR consists of two main components: an encoder and a decoder. The encoder leverages RoBERTa to encode the question and passage into contextual embeddings. The decoder, which consists of four modules (RM Module, Selector, MA Module, and Program Solver), generates inference steps and answers based on contextual information from the encoder. We implement our model using PyTorch and evaluate it on the FINQA dataset, a benchmark for numerical reasoning in the financial domain. Experimental results demonstrate that RMMANR outperforms several baseline models, achieving superior accuracy.

1. Introduction

Numerical reasoning refers to the ability to understand and work with numbers, make sense of quantitative information, and perform mathematical operations. In numerical reasoning, a person or machine solves a question that typically involves the use of numerical data in a given document. Numerical reasoning encompasses a range of mathematical operations such as arithmetic, algebra, ratios, etc. While simple numerical reasoning tasks, such as math word problems (MWPs) [1], typically involve a few arithmetic operations, more complex tasks—such as those in the financial domain—require analyzing heterogeneous data sources like financial reports to answer intricate questions and the solution [2,3]. In healthcare, numerical reasoning aids in interpreting patient data (such as medical images and clinical readings), diagnosing conditions, and making treatment decisions. Similarly, information technology may involve extracting patterns and insights from large datasets [4].

An example of numerical reasoning in the financial domain is illustrated in Figure 1. Here, the task is to determine the percentage change in cash flow hedges from 2010 to 2011 based on a given passage. Solving such problems requires identifying relevant operands and deducing the correct operators from the passage, based on the question. Even competitive models like ChatGPT-4 struggle with numerical reasoning exams [5]. FINQA is the first dataset of its kind that handles complex QA tasks based on real financial documents. It consists of 8281 question–answer pairs, collaboratively constructed by 11 finance professionals using earnings reports from $S\&P$ 500 companies. Answering questions in the FINQA dataset involves many common financial analysis calculations, such as addition, subtraction, multiplication, division, averaging, summation, comparison and so on [2].

The encoder–decoder architecture has emerged as a widely adopted deep neural network architecture for solving numerical reasoning tasks due to its ability to derive mathematical equations from textual inputs. As depicted in Figure 2, this architecture typically employs an encoder to transform a question and a document into vector representations, which are then processed by a decoder to produce the corresponding equation. More recent models, such as FINQA [2], ELASTIC [6], and SoarGraph [7], leverage pre-trained language models like RoBERTa to encode problems and documents, significantly improving performance. In terms of the decoder, early approaches, such as the deep neural solver (DNS) [8], utilized LSTMs, while later advancements introduced tree decoders to generate prefix traversal sequences of expression trees [9]. In the financial domain, FINQA proposes a retrieval-based generator QA framework that first retrieves supporting facts from financial reports and then generates executable reasoning programs to answer questions [2].

Despite these advancements, two significant challenges persist in numerical reasoning: (1) information loss as network depth increases, leading to the gradual dissipation of embedding vectors for questions and passages, and (2) suboptimal performance in multi-step reasoning tasks, leaving substantial room for accuracy improvement. To address these issues, we propose a model with a recall mechanism and multi-head attention for numerical reasoning (RMMANR). RMMANR consists of two main components: an encoder and a decoder. The encoder leverages RoBERTa to encode the question and passage into contextual embeddings. The decoder, which consists of four modules (RM Module, Selector, MA Module, and Program Solver), generates reasoning steps and answers based on contextual information from the encoder. The main contribution of our work is three-fold:

• We propose a recall mechanism for numerical reasoning, which refocuses on the embedding information of questions and passages before generating operators and operands. This mechanism prevents the loss of critical information as the model deepens, ensuring that the initial input embeddings are retained throughout the network.
• We propose multi-head attention for numerical reasoning, which enhances the model’s ability to handle complex, multi-step reasoning tasks. By efficiently extracting multiple relevant information fragments, this mechanism optimizes the solution process for multi-step problems.
• We evaluate our proposed model, RMMANR (recall mechanism and multi-head attention for numerical reasoning), on the FINQA dataset [2]. Experiment results show that RMMANR outperforms several baseline models, achieving superior accuracy and robustness.

2. Related Works

In this section, we review existing research on numerical reasoning, focusing on three key aspects: datasets, question and document representation, and reasoning modules.

2.1. Datasets

Numerous datasets have been developed to evaluate numerical reasoning capabilities. Early Math23K consists of 23,161 Chinese math word problems curated from online educational platforms and refined using rule-based extraction methods [10]. DROP, proposed by Dua et al., emphasizes discrete reasoning over paragraphs and requires more complex arithmetic operations [11]. Recently, domain-specific datasets like FINQA [2] and TAT-QA [12,13] have been introduced. FINQA contains 8281 financial question–answer pairs derived from S&P 500 company reports, while TAT-QA features 16,552 questions based on 2757 hybrid contexts of tables and textual descriptions. These datasets focus on numerical reasoning in financial domains and support the development of explainable models.

2.2. Question and Document Representation

Early approaches to numerical parsing relied on language parsers, such as the Stanford CoreNLP suite, to identify variables, operands, and their relationships in questions and documents [14,15]. With the rise of deep learning, word embeddings became the dominant method for representing questions and documents. Early embeddings, such as skip-gram, CBOW, and GloVe, used static word vectors, where each token was represented by a single vector [16,17,18]. However, these approaches struggled with polysemy, as tokens like words can have multiple meanings.

To address this limitation, contextualized word embeddings were introduced. Early efforts, such as ELMo, used bidirectional LSTMs to generate context-aware embeddings [19]. The advent of transformer-based models, such as BERT, RoBERTa, and LLaMA [20], revolutionized the field by enabling dynamic, context-sensitive representations. These models have been widely adopted in numerical reasoning tasks. For example, neural-symbolic reader (NeRd) uses BERT as its encoder [21], while models like QDGAT [22], MHST [23], and ELASTIC [6] leverage RoBERTa for encoding questions and documents. Additionally, models like LayoutLM and LayoutLMv2 [24] extend these approaches to visually rich documents by incorporating text, image, and layout information. When the reasoning process involves images within a document, certain image feature techniques can also be applied. For example, Li et al. use the combined channel and spatial attention (CCSA) mechanism, which emphasizes capturing both spatial and channel-wise features while effectively preserving essential high-frequency details [25].

2.3. Reasoning Module

The reasoning module in deep neural networks typically incorporates a decoder, which can be based on feed-forward neural networks (FFNNs), recurrent neural networks (RNNs), multi-head attention, transformers, or graph neural networks (GNNs).

Table 1. Performance of deep neural network approaches in different numerical reasoning datasets. Exe Acc (execution accuracy) measures whether the executed result of the generated program matches the ground truth answer. Prog Acc (program accuracy) evaluates whether the generated program is mathematically equivalent to the ground truth program. The F1 Score assesses the overlap between the predicted and gold answers using a bag-of-words representation. Bold values indicate the best performance for each metric within a given dataset.

Category	Model	Math23K	DROP		FINQA		TAT QA
		Prog Acc	Exu Acc	F1	Exu Acc	Prog Acc	Exu Acc	F1
	Human Expert [2]	-	-	-	91.16	87.49	-	-
FF	MHST [23]	-	-	-	-	-	63.6	72.7
	FinMath [26]	-	-	-	-	-	58.3	68.2
RNN	DNS [10]	64.7	-	-	-	-	-	-
	NeRd [21]	78.55	81.85	52.48	49.9	-	-	-
	ELASTIC [6]	-	-	-	68.96	65.21	-	-
Transformer	Ensemble [27]	66.7	-	-	-	-	-	-
Graph	ODGAT [22]	-	64.56	67.97	13.1	-	39.1	49.7
	SoarGraph [7]	-	-	-	67.2	-	65.4	75.3

summarizes the performance of various approaches.

Feed-Forward Neural Networks: Models like MHST use multi-head classifiers to identify arithmetic operations and relevant numbers from questions and documents [23]. GDS introduces a goal-driven approach to generate expression trees by recursively decomposing target outcomes into subgoals [9]. FinMath adopts a similar strategy for financial numerical reasoning [26].

Recurrent Neural Networks (RNNs): The deep neural solver (DNS) combines an encoder–decoder architecture with a similarity-based retrieval model, achieving accuracies of 64.7% and 70.1% on Math23K and Alg514, respectively [10]. NeRd integrates BERT for encoding and an LSTM-based programmer for generating multi-step reasoning programs [21]. ELASTIC employs a GRU-based decoder with an attention layer and a memory register, achieving 68.96% execution accuracy on the FINQA dataset [6].

Transformers: Transformer-based decoders, such as those used by Wang et al., generate mathematical expression trees but often underperform compared to LSTM-based decoders [27]. Ensemble models combining multiple encoder–decoder architectures have shown improved results on datasets like Math23K.

Graph Neural Networks (GNNs): Doc2SoarGraph converts embeddings into hierarchical graphs (e.g., quantity comparison, text relation, and date comparison graphs) and processes them using a graph convolutional neural network (GCNN) [7]. QDGAT constructs graphs from questions and documents, using a question-directed layer to simulate reasoning processes [22].

Prompting: Prompting is a technique used to guide large language models (LLMs) to generate more accurate and structured responses without modifying the model’s parameters. By carefully designing the input query, users can significantly enhance the model’s reasoning ability in numerical tasks. There are several strong prompting techniques that improve LLM performance in numerical reasoning. Chain-of-thought (CoT) [28] demonstrates how step-by-step reasoning enhances model performance. Xuezhi Wang et al. proposed self-consistency in “Self-Consistency Improves Chain of Thought Reasoning in Language Models”, where multiple reasoning paths are sampled to select the most consistent answer [29]. Program-aided language (PAL) models explore integrating code execution with language models, enabling precise calculations by generating and running code [30].

Despite these advancements, challenges such as information loss in deep networks and suboptimal multi-step reasoning in numerical reasoning remain significant. Inspired by ELASTIC [6], we propose a recall mechanism and multi-head attention for numerical reasoning (RMMANR) to address these limitations.

3. Methods

Deep neural networks often suffer from the gradual loss of embedding vectors for questions and passages as the network depth increases. To address this issue, we propose a recall mechanism for numerical reasoning, designed to mitigate the loss of embedding vectors for questions and passages as network depth increases. In memory and cognitive psychology, a recall mechanism refers to the process by which the brain retrieves stored information from long-term memory. This mechanism allows individuals to access past experiences, facts, or learned skills. Inspired by the human tendency to review relevant information before making a decision, RM incorporates a recall module that revisits question and passage embeddings before generating operators and operands. Unlike most models, where the Reasoning Module directly passes the guidance vectors—obtained from embedding vectors of questions and passages—to the Generator Module for operator and operand generation, RM introduces an additional recall step. Specifically, as shown in Figure 3, the Recall Module is placed between the Reasoning Module and the Generator Module, enabling second access to question and passage embeddings. This additional retrieval step ensures that the guidance vectors contain richer contextual information, leading to more accurate operator and operand generation. By integrating this mechanism, RM prevents the degradation of initial input embeddings, enhancing numerical reasoning accuracy.

On the other hand, many existing models struggle with multi-step reasoning tasks [2,6]. To address this challenge, we propose multi-head attention for numerical reasoning. As shown in Figure 4, the MA Module consists of a multi-head attention reasoning module and a Generator Module. Multi-head attention enables the model to focus on different parts of the input sequence simultaneously, facilitating parallel processing of multiple reasoning steps. This capability is particularly beneficial for complex numerical reasoning tasks, as it efficiently extracts relevant information fragments and optimizes the solution process for multi-step problems.

RMMANR combines the strengths of the RM and MA Modules. As illustrated in Figure 5, the question and passage embeddings are first processed by the RM Module, which generates initial operators $O{P}_{RM}$ and operands $O{E}_{RM}$. The Selector then calculates $stepNum$ based on $O{P}_{RM}$. If $stepNum=1$, the Selector passes $O{P}_{RM}$ and $O{E}_{RM}$ to the Program Solver. Otherwise, the MA Module is activated to generate $O{P}_{MA}$ and $O{E}_{MA}$, which are then passed to the Program Solver for execution.

3.1. Task Definition

All the notations are defined in

Table 2. Task definition notation.

Notation	Description
P, Q, R	The text of the passage, the text of the question, the program of numerical reasoning.
$NUM$	The numbers that appear in Q and P.
$CONS$	Constants described in domain-specific language (DSL).
$OP$	All mathematical operators.
$o{p}_i$	The i-th mathematical operator in R.
$OE$	All operands.
$o{e}^i$	All operands associated with $o{p}_i$.
$o{e}_j^i$	The j-th operand of $o{p}_i$.
$O{P}_{RM}$	An array of mathematical operators generated by RM Module.
$O{P}_{MA}$	An array of mathematical operators generated by MA Module.
$O{E}_{RM}$	An array of mathematical operands generated by RM Module.
$O{E}_{MA}$	An array of mathematical operands generated by MA Module.
s	Selected from either $OP$ or $OE$, R consists of s.
$r_i$	$r_i=o{p}_i\left[o{e}^i\right]$, the i-th subprogram of R.
$e_s$	Symbol s’s embedding from the decoding vocabulary.
$E_{op}\left(s\right)$	The embedding retrieval function of $OP$.
$E_{cons}\left(s\right)$	The embedding retrieval function of $CONS$.
$E_{num}\left(s\right)$	The embedding retrieval function of $NUM$.

, and these definitions align with those used in ELASTIC [6]. Given a question Q and a passage P, the task is to construct a numerical reasoning program R, which consists of a sequence of symbols s (operators $OP$ and operands $OE$). Operands are derived from constants $CONS$ or numbers $NUM$ in the input. The program R is structured as:

$$R={\left\{o{p}_i{\left[o{e}_j^i\right]}_{j=0}^{m-1}\right\}}_{i=0}^{n-1}$$

(1)

where $o{p}_i$ is the i-th operator and $o{e}_j^i$ are its associated operands. The program R represents the reasoning process and can be executed to derive the final answer. As shown in Appendix A, it is an example of financial question–answering problems, using nations defined in Table 2.

3.2. Proposed RMMANR

As illustrated in Figure 5, our proposed model, RMMANR, consists of two main components: an encoder and a decoder. The encoder transforms the question and passage into contextual embeddings. The decoder comprises four modules: RM Module, Selector, MA Module, and Program Solver. The process begins with the question and passage embeddings being fed into the RM Module, which generates initial operators and operands. The Selector then determines whether to activate the MA Module based on the complexity of the task (i.e., the number of reasoning steps). Finally, the Program Solver executes the generated program to produce the final result.

3.3. Encoder

As shown in Figure 5, the encoder in RMMANR is implemented using RoBERTa, which transforms the combined sequence of the question Q and passage P into contextual embeddings h. Specifically, $h^q$ represents the embedding of the question. These embeddings capture the semantic and contextual information necessary for numerical reasoning. The decoder then uses h to generate the numerical reasoning program R and derive the final result.

RMMANR is encoder-agnostic, meaning that any model capable of providing sequence-context vectors, such as BERT, RoBERTa, or LLaMa, can be used as the encoder. We chose RoBERTa primarily due to its efficiency in memory usage and its competitive performance in capturing contextual relationships, although other models with similar functionality could also be applied.

3.4. Decoder

As shown in Figure 5, the decoder in RMMANR is responsible for generating the numerical reasoning program R based on the contextual embeddings h provided by the encoder. It consists of four modules: RM Module, Selector, MA Module, and Program Solver. The decoding process begins with the RM Module, which generates initial operators and operands, followed by the Selector determining whether to activate the MA Module for multi-step reasoning. Finally, the Program Solver executes the generated program to produce the final result.

3.4.1. Decoding Vocabulary and Token Embedding

The decoding vocabulary consists of two primary elements: operators $OP$ and operands $OE$. Operands are further divided into constants $CONS$ (e.g., mathematical constants like $\pi$) and numbers $NUM$ extracted from the input. The embedding retrieval function $E_{op,cons,num}\left(s\right)$ maps a symbol s in the decoding vocabulary to its corresponding embedding $e_s$. Specifically, the embedding for a symbol s is defined as:

$$e_s=\left\{\begin{array}{cc}{E}_{op}\left(s\right)\hfill & \hfill \mathrm{if}s\in OP,\\ E_{cons}\left(s\right)\hfill & \hfill \mathrm{if}s\in CONS,\\ E_{num}\left(s\right)=h_i\hfill & \hfill \mathrm{if}s\in NUM,\end{array}\right.$$

(2)

where $E_{op}\left(s\right)$, $E_{cons}\left(s\right)$, and $E_{num}\left(s\right)$ are the embedding retrieval functions for operators, constants, and numbers, respectively.

3.4.2. RM Module

The RM Module applies a recall mechanism to refocus on the embedding information from the question and passage before generating operators and operands, ensuring the initially inputted embeddings are not lost as the model deepens. As illustrated in Figure 3, the RM Module is composed of three modules: Reasoning Module, Recall Module, and Generator Module. The Recall Module receives initial guidance vectors $g_{initial\_op}$ from the Reasoning Module, recalls relevant information $h^q$, and outputs $g_{final\_op}$, which directs the operator generator to produce the operator $o{p}_t$. Concurrently, the Recall Module receives initial guidance vectors $g_{initial\_oe}$ from the Reasoning Module, recalls relevant information h, and outputs $g_{final\_oe}$. Next, the operator generator pauses and passes $g_{final\_op}$ and $o{p}_t$ to the operand generator, which generates the first operand $o{e}_t^1$. Once the operator and operands of the subprogram $r_t$ are completed, the RM Module then moves on to generate the the operator and operands of the next subprogram $r_{t+1}$. After the array of mathematical operators $O{P}_{RM}$ and the array of mathematical operators generate $O{E}_{RM}$, they are passed to the Selector.

Reasoning Module: As shown in Figure 5, the inputs for the Reasoning Module are the embedding of the previously generated symbol $s_{t-1}$ and contextual vector $h^q$ from the encoder. The Reasoning Module first calculates the context vector c by the attention weights $a_i$ and the normalized vectors of $h_i^q$:

$$c=\sum _i{a}_i{h}_i^q$$

(3)

$$a_i={\displaystyle \frac{exp\left(score(e_{s_{t-1}},h_i^q)\right)}{{\sum }_{j}exp\left(score(e_{s_{t-1}},h_j^q)\right)}}$$

(4)

$$score(e_{s_{t-1}},h_i^q)=e_{s_{t-1}}^T{W}_1{W}_2{h}_i^q$$

(5)

where $W_1\in {\mathbb{R}}^{h*h}$, $W_2\in {\mathbb{R}}^{h*h}$, $h=1024$, and both weight parameters are trainable. The encoding information from the encoder is summarized into c based on the previously generated symbol $s_{t-1}$. Next, the Reasoning Module uses the GRU [31] to obtain the initial guidance output $g_{initial\_op}$:

$$g_{initial\_op},H_t=GRU(Relu\left(W_3[c:E_{op,cons,num}\left(s_{t-1}\right)]\right),H_{t-1})$$

(6)

where “:” denotes concatenation. $Relu$ is the activation function, and $W_3\in {\mathbb{R}}^{h*2h}$ is a trainable parameter. $H_{t-1}\in {\mathbb{R}}^h$ represents the GRU’s hidden state from the previous step, and $H_0$ is initialized to zero. The Reasoning Module outputs the vector $g_{initial\_op}$. The process of obtaining $g_{initial\_oe}$ is the same as that of obtaining $g_{initial\_op}$, with the main difference being that $g_{initial\_oe}$ is obtained based on h. Though $g_{initial\_op}$ can guide the Generator Module to produce $op$ and $oe$. In the RM Module, $g_{initial\_op}$ is not used directly to generate operators or operands.

Recall Module: As the network deepens, the information from input questions and passages tends to dissipate in many models. Similar to how humans solve math problems by revisiting the question and passage for verification, the RM Module applies a recall mechanism to refocus on the embedding information before generating operators and operands. This ensures that important question and passage information is retained throughout the model’s depth. When predicting operators, the Recall Module recalls the contextual vector $h^q$. $h^q$ is the key/K of the concern layer, and $g_{initial\_op}$ is the query/Q of the concern layer. The Recall Module calculates the final guidance vector $g_{final\_op}$ by the attention weights $b_i$ and the normalized vectors of $h_i^q$:

$$g_{final\_op}=\sum _i{b}_i{h}_i^q$$

(7)

$$b_i={\displaystyle \frac{exp\left(score(g_{initial\_op},h_i^q)\right)}{{\sum }_{j}exp\left(score(g_{initial\_op},h_j^q)\right)}}$$

(8)

$$score(g_{initial\_op},h_i^q)=g_{initial\_op}^T{W}_4{W}_5{h}_i^q$$

(9)

where $W_4\in {\mathbb{R}}^{h*h}$, $W_5\in {\mathbb{R}}^{h*h}$, $h=1024$, and both weight parameters are trainable. The $g_{final\_op}$ recalls the embedding vectors from the encoder based on the $g_{initial\_op}$. The process of obtaining $g_{final\_oe}$ is the same as that of obtaining $g_{final\_op}$, with the main difference being that $g_{final\_oe}$ is obtained based on h.

Generator Module: The Generator Model consists of an operator generator and an operand generator. The guidance vector $g_{final\_op}$ enters the operator generator. The operator generator uses softmax to calculate the most likely i-th operator:

$$P\left(o{p}_i\right|g_{final\_op},E_{o{p}_{t-1}})=softmax\left(E_{o{p}_i}^T{R}_{final\_op}\right)$$

(10)

$$R_{final\_op}=Relu\left(W_{op}{g}_{final\_op}\right)$$

(11)

The guidance vector $g_{final\_oe}$ enters the operand generator. The operand generator uses softmax to calculate the most likely i-th operand:

$$P\left(o{e}_i\right|g_{final\_oe},E_{o{e}_{i-1}})=softmax\left(E_{o{e}_i}^T{R}_{final\_oe}\right)$$

(12)

$$R_{final\_oe}=Relu\left(W_{argu}{g}_{final\_oe}\right)$$

(13)

3.4.3. MA Module

Many current models struggle with multi-step reasoning tasks [2,6]. Multi-head attention allows the model to focus on different parts of the input sequence simultaneously. This enables parallel processing of multiple reasoning steps, which is especially beneficial for complex numerical reasoning tasks. By analyzing different possible solutions, multi-head attention optimizes the solution process for multi-step problems. As shown in Figure 4, the MA Module consists of two modules: a Multi-head Attention Reasoning Module and a Generator Module. The Multi-head Attention Reasoning Module processes the contextual embeddings, and the Generator Module generates the array of mathematical operators $O{P}_{MA}$, and the array of mathematical operators generates $O{E}_{MA}$.

Multi-head Attention Reasoning Module: The input to the Multi-head Attention Reasoning Module consists of the previously generated symbol $s_{t-1}$ and the embedded context vector h from the encoder. The reasoning module first calculates the context vector c using the multi-head attention mechanism, which allows it to effectively focus on different segments of the input. In the multi-head attention mechanism, attention weights and context vectors are calculated separately for each attention head. The specific steps are as follows:

• Calculate the attention weights for each attention head: For the kth attention head, calculate the attention weight $a_i^{\left(k\right)}$ from the previously generated symbol $s_{t-1}$ and the context vector h:

  $$a_i^{\left(k\right)}={\displaystyle \frac{exp\left(score(e_{s_{t-1}},h_i^{\left(k\right)})\right)}{{\sum }_{j}exp\left(score(e_{s_{t-1}},h_j^{\left(k\right)})\right)}}$$

  (14)

  where the scoring function for the kth attention head is:

  $$score(e_{s_{t-1}},h_i^{\left(k\right)})=e_{s_{t-1}}^T{W}_6^{\left(k\right)}{W}_7^{\left(k\right)}{h}_i^{\left(k\right)}$$

  (15)
  Here, $W_6^{\left(k\right)}\in {\mathbb{R}}^{h*h}$ and $W_7^{\left(k\right)}\in {\mathbb{R}}^{h*h}$ are trainable parameters of the kth attention head, where $h=1024$.
• Calculate the context vector for each attention head: Different heads provide diverse attention perspectives, enabling the model to focus on multiple levels of information simultaneously, thereby enhancing its modeling capability. Sum the weighted vectors of attention heads to obtain the context vector $c^{\left(k\right)}$ for the kth attention head:

  $$c^{\left(k\right)}=\sum _i{a}_i^{\left(k\right)}{h}_i^{\left(k\right)}$$

  (16)
• Merge the context vectors of multiple attention heads: Combine the context vectors $c^{\left(k\right)}$ from all attention heads and obtain the final context vector c through a linear transformation:

  $$c=W_c\left[c^{\left(1\right)},c^{\left(2\right)},\dots ,c^{headNum}\right]$$

  (17)

  where $W_c$ is the linear transformation matrix after merging, and
  $headNum=4$ is the number of attention heads.

Next, the reasoning module uses a GRU to compute the initial guided output $g_{initial\_op}$:

$$g_{initial\_op},H_t=GRU(\mathrm{Relu}\left(W_8[c:E_{op,cons,num}\left(s_{t-1}\right)]\right),H_{t-1})$$

(18)

where “:” denotes concatenation, Relu is the activation function, and $W_8\in {\mathbb{R}}^{h*h}$ is a trainable parameter. $H_{t-1}\in {\mathbb{R}}^{h*h}$ represents the hidden state of the GRU in the previous step, and $H_0$ is initialized to zero.

Generator Module: In the MA Module, $g_{initial\_op}$ directly guides the Generator Module to generate operators and operands. The Generator Model consists of an operator generator and an operand generator. The guidance vector $g_{initial\_op}$ enters the operator generator. The operator generator uses softmax to calculate the most likely i-th operator:

$$P\left(o{p}_i\right|g_{initial\_op},E_{o{p}_{t-1}})=softmax\left(E_{o{p}_i}^T{R}_{initial\_op}\right)$$

(19)

$$R_{initial\_op}=Relu\left(W_{op}{g}_{initial\_op}\right)$$

(20)

The guidance vector $g_{initial\_oe}$ enters the operand generator. The operand generator uses softmax to calculate the most likely i-th operand:

$$P\left(o{e}_i\right|g_{initial\_oe},E_{o{e}_{i-1}})=softmax\left(E_{o{e}_i}^T{R}_{initial\_oe}\right)$$

(21)

$$R_{final\_oe}=Relu\left(W_{argu}{g}_{initial\_oe}\right)$$

(22)

3.4.4. Selector

Selector uses the hyperparameter $stepNum$ as a selection threshold and calculates $stepNum$ by the size of $O{P}_{RM}$ generated by the RM Module. If $stepNum$$=1$, the Selector chooses to pass the $O{P}_{RM}$ and $O{E}_{RM}$ generated by the RM Module to the Program Solver. If $stepNum>1$, the Selector starts the MA Module, which then generates $O{P}_{MA}$ and $O{E}_{MA}$ and passes them to the Program Solver.

3.4.5. Program Solve

Program Solve is used to calculate programs. Since subsequent subprograms may use the result of the previous subprogram as an operand, we store the result of the previous subprogram in #n of a memory registry like ELASTIC [6].

3.5. RMNR and MANR

For the framework of the model RMNR, see Appendix B. The main difference between RMNR and RMMANR is that RMNR does not have an MA Module and Selector. For the framework of the model MANR, see Appendix C. The main difference between MANR and RMMANR is that MANR does not have an RM Module and Selector.

4. Experiments

4.1. Dataset

The FINQA dataset [2] consists of 8281 question–answer pairs derived from annual financial reports. The dataset is divided into training (6251 samples), evaluation (883 samples), and test (1147 samples) sets. Each sample includes a financial question, a supporting passage, and a gold-standard numerical reasoning program. The FINQA dataset is designed to test models’ ability to perform complex numerical reasoning in the financial domain. We evaluate our model on the processed FINQA dataset from ELASTIC. ELASTIC has already converted all table data related to the reasoning process into text and placed it in the passage during the data preprocessing stage (The preprocessed FINQA dataset is available at https://github.com/NeuraSearch/NeurIPS-2022-Submission-3358 (accessed on 15 July 2022).)

4.2. Evaluation Metrics

Following the original FINQA paper, we use two evaluation metrics: execution accuracy (Exe Acc) and program accuracy (Prog Acc).

• Execution Accuracy measures the correctness of the final answer produced by executing the model-predicted program against the gold-standard executable result.
• Program Accuracy evaluates the precision of the predicted program by comparing the operands and operators in the model-generated program with those in the gold-standard program.

These metrics provide a comprehensive assessment of both the reasoning process (program accuracy) and the final result (execution accuracy).

4.3. Baselines

We use several prompting models and fine-tuning models to compare with RMNR, MANR, and RMMANR.

Prompting models include the following:

• GPT-3.5-turbo: GPT-3.5-turbo is a large language model with 175 billion parameters [32].
• GPT-4: GPT-4 is a large multimodal model that is able to process both text and image inputs and generate text outputs [33].
• Program-of-Thought: Program-of-Thought first generates programming and text statements, and then produces an answer [28].

Fine-tuning models include:

• FINQANet: FINQANet employs a typical encoder–decoder architecture, where pre-trained LMs are used as the encoder, and LSTM serves as the decoder [2].
• NeRd: NeRd generates symbolic nested programs using BERT and a model based on pointer generators [21].
• NumNet: NumNet models arithmetic information through GNN networks [34].
• ELASTIC: ELASTIC separately generates mathematical operators and operands, reducing the occurrence of cascading errors [6]. (The preprocessed FINQA dataset and code are available at https://github.com/NeuraSearch/NeurIPS-2022-Submission-3358 (accessed on 15 July 2022).)
• DyRRen: DyRRen is an extended retriever-reranker-generator framework [35].
  We also compared the reasoning results of non-experts and experts from the original FINQA paper [2].

4.4. Implementation Details

RMMANR is trained on a server with an RTX4090 GPU with 24GB of memory, implemented by Transformer 4.44.0 [36] and Pytorch 2.3.0 [37]. The training batch size was set to 3 and the epoch to 100. We chose Adam as the optimizer [38] to perform the update of the model parameters. The initial learning rate was set to $1\times {10}^{-5}$, and the learning rate was halved every 25 epochs. In order to prevent overfitting, during training, the dropout rate was set to 0.1, and the weight decay was set to $1\times {10}^{-5}$.

5. Experiment Results and Analysis

5.1. Overall Results

Table 3. Overall results of our models and the baselines on the test data. † denotes that the results are sourced from the original paper. ‡ denotes the results are sourced from [2]. * denotes that the results are sourced from [6]. ! denotes that the results are sourced from [39]. Bold values indicate the best performance for each metric.

Category	Models	FINQA Results
		Exe Acc (%)	Prog Acc (%)
Prompting-Model	GPT-3.5-turbo [32] !	48.56	-
	GPT-4 [33] !	68.79	-
	Program-of-Thought [28] !	68.10	-
Fine-Tuning-Model	NumNet [34] *	2.32	-
	NumNet+ [34] *	10.29	-
	NeRd [21] ‡	52.48	49.90
	FINQANet (RoBERTa-base) [2] †	60.10	58.38
	FINQANet (RoBERTa-large) [2] †	65.05	63.52
	ELASTIC (RoBERTa-base) [6] †	62.66	59.28
	ELASTIC (RoBERTa-large) [6] †	68.96	65.21
	DyRRen [35] !	63.30	61.29
	RMNR (RoBERTa-base)	65.33	61.20
	RMNR (RoBERTa-large)	70.58	66.61
	MANR (RoBERTa-base)	64.98	61.12
	MANR (RoBERTa-large)	70.21	65.74
	RMMANR (RoBERTa-large)	71.63	67.31
Human Performance	Human Expert †	91.16	87.49
	Human Non-Expert †	50.68	48.17

compares the performance of our models (RMNR, MANR, and RMMANR) with several baselines on the FINQA dataset. Notably, all three of our models outperform the existing baselines when using RoBERTa as the encoder. Specifically, RMMANR (RoBERTa-large) achieves the highest performance, surpassing all baselines by at least 2.67% in execution accuracy and 2.1% in program accuracy. Compared to FINQANet (RoBERTa-large) from the original FINQA paper, RMMANR improves execution accuracy by 6.58 points and program accuracy by 3.79 points.

Both RMNR (RoBERTa-large) and MANR (RoBERTa-large) also demonstrate significant improvements over ELASTIC (RoBERTa-large). RMNR increases execution accuracy by 1.62 points and program accuracy by 1.4 points, while MANR improves execution accuracy by 1.25 points and program accuracy by 0.53 points. Given that RMNR, MANR, and ELASTIC use RoBERTa encoders of the same size, these results highlight the effectiveness of the recall mechanism and multi-head attention in enhancing numerical reasoning capabilities.

Our models also outperform other baselines, such as NeRd, NumNet, and DyRRen. Unlike NeRd, which relies on external rules for operators [34], our models achieve superior results without such dependencies. Additionally, NumNet and NumNet+ exhibit poor scalability due to limitations in their internal architectures [40], while DyRRen’s retriever–reranker–generator framework is prone to cascading errors, resulting in lower performance.

Furthermore, our models outperform all prompting models listed in Table 3, demonstrating stronger reasoning abilities compared to models like GPT-3.5-turbo and GPT-4. While RMNR, MANR, and RMMANR surpass non-expert human performance, there remains a significant gap between our models and expert human performance. This suggests that there is still considerable room for improvement on the FINQA dataset.

5.2. The Convergence Process

The convergence process of RMNR (RoBERTa-large) is illustrated in Figure 6. RMNR demonstrates rapid convergence during the initial five training rounds, capturing the majority of core data features and patterns. In subsequent training iterations, the model’s accuracy continues to improve steadily, indicating that it is fine-tuning its parameters to better fit the data by learning more intricate features. However, occasional fluctuations in accuracy are observed in later stages, which may be attributed to the presence of noisy data. The convergence rate slows significantly towards the end of training, with the highest accuracy achieved near the final iterations. This suggests the potential for further optimization of hyperparameters to enhance performance.

The convergence behavior of MANR (RoBERTa-large), as depicted in Figure 7, is similar to that of RMNR (RoBERTa-large) and is therefore not discussed in detail here.

It is worth noting that execution accuracy may overestimate the model’s performance, as it can occasionally produce correct answers by chance. On the other hand, program accuracy may lead to false negatives in cases where a question admits multiple valid reasoning programs. Since correct reasoning programs invariably yield correct results, execution accuracy is generally higher than program accuracy.

5.3. Performance on Different Program Steps

The models’ performance across different program steps is presented in

Table 4. Results on different program steps. † denotes that the results are sourced from the original ELASTIC [6] paper. Each model employs RoBERTa-large for encoding. Bold values indicate the best performance for each metric.

Model	The Entire Dataset		=1(3717)		=2(3013)		≥3(512)
	Eex Acc	Prog Acc	Eex Acc	Prog Acc	Eex Acc	Prog Acc	Eex Acc	Prog Acc
FINQANet †	65.05	63.52	73.70	71.25	62.34	59.65	28.57	23.80
ELASTIC †	68.96	65.21	76.30	75.66	66.01	66.01	31.78	31.10
RMNR	70.58	66.61	78.96	76.61	65.04	58.68	32.14	27.38
MANR	70.21	65.74	76.49	74.01	68.22	60.88	30.95	25.00
RMMANR	71.63	67.31	80.34	77.68	66.01	59.41	30.95	25.00

. When the program step is 1, RMNR outperforms ELASTIC, FINQANet, and MANR, demonstrating its superior capability in handling single-step program scenarios. MANR achieves the best performance on two-step reasoning tasks, suggesting that its multi-head attention mechanism is particularly effective for reasoning tasks of moderate complexity. RMMANR exhibits the highest performance on both the entire dataset and single-step programs, while also surpassing RMNR on two-step programs. This indicates that RMMANR successfully integrates the strengths of RMNR and MANR, leveraging their complementary features.

Interestingly, all models experience a significant drop in accuracy—approximately halved—when transitioning from two-step to more than two-step reasoning tasks. As shown in Table 4, the number of training examples with more than two program steps (521) is substantially smaller compared to those with one (3717) or two (2013) steps. We hypothesize that the models’ poor performance on tasks with more than two steps is primarily attributable to the limited number of training examples available for such scenarios. Consequently, the lack of sufficient training data for longer program steps is likely the main factor contributing to the observed performance degradation. This issue highlights an important direction for future research, particularly in addressing data scarcity for complex multi-step reasoning tasks.

5.4. Error Analysis

To better understand the limitations of our model in predicting mathematical operations, we analyze common error patterns. The errors can be categorized into the following three types:

Operator prediction error: One common source of error is the incorrect selection of mathematical operations. For instance, as shown in Example 1 in Figure 8, in a case where the task required computing the percentage change in deferred tax assets, the correct sequence involved a subtraction operation, but the model mistakenly predicted an addition operation. Such errors often arise when the model fails to grasp the underlying mathematical relationship, particularly in scenarios involving changes or differences where subtraction is required instead of addition.

Operand prediction error: Another prevalent issue is the misidentification of numerical values within the input text. As shown in Figure 8, in Example 2 involving the ratio of collateral pledged to short-term borrowings, the model incorrectly selected 19.3 instead of the correct value 2.3. This type of error may stem from: (1) Numerical mismatch: The model might prioritize selecting numbers that appear more prominently in the text, overlooking the correct values required by the problem. (2) Contextual misunderstanding: The model may fail to correctly associate numerical values with temporal constraints, leading to the selection of numbers from incorrect time periods or categories.

Operand and operator prediction error: As shown in Example 3 in Figure 8, in a case requiring the calculation of the percentage change in the fair value of non-vested shares, the correct sequence should involve a multiplication step followed by subtraction and division. However, the model only predicted multiplication and division, neglecting the intermediate subtraction operation. This suggests that the model struggles with multi-step reasoning, particularly in percentage-based calculations, where computing the absolute difference before division is crucial.

These errors indicate three major challenges: (1) inaccurate selection of mathematical operations, (2) incorrect numerical value extraction, and (3) incorrect operands and operators. To mitigate these issues, future work can focus on the following:

• Enhancing the model’s ability to understand mathematical relationships by refining operation selection mechanisms.
• Improving number extraction accuracy through better numerical alignment techniques.
• Strengthening multi-step reasoning to ensure all steps are correctly executed.

5.5. Ablation Studies

As illustrated in Figure 9 and Figure 10, both execution accuracy and program accuracy of RMRN and MARN are significantly higher than those of RMMARN without the recall mechanism and multi-head attention. This indicates that either the recall mechanism or multi-head attention can effectively enhance the model’s accuracy. Furthermore, RMMARN achieves the highest accuracy, demonstrating that the combination of these two components enables the model to attain optimal performance. For the framework of RMMARN without the recall mechanism and multi-head attention, refer to Appendix D.

Figure 11 and Figure 12 reveal that RMNR with both operator and operand recall achieves the highest scores. Notably, RMNR with only operand recall performs worse than RMNR without any recall modules. This degradation in performance may be attributed to the recalled h containing excessive numerical noise. Since the model must select the correct operand from a large set of numbers, revisiting h before performing operand prediction could distract the model and increase the difficulty of identifying the correct operand. This observation highlights a potential area for future research.

Figure 13 and Figure 14 demonstrate that MANR with four attention heads achieves the best performance, both on the entire FINQA dataset and on multi-step reasoning tasks within FINQA. This suggests that, for MANR, increasing the number of attention heads does not necessarily improve performance. Instead, the configuration of four heads is optimal for this model.

6. Conclusions and Future Work

This paper introduces the recall mechanism and multi-head attention for numerical reasoning (RMMANR). RMMANR consists of two main components: an encoder and a decoder. The encoder leverages RoBERTa to encode the question and passage into contextual embeddings. The decoder, which consists of four modules (RM Module, Selector, MA Module, and Program Solver), generates inference steps and answers based on contextual information from the encoder. The recall mechanism addresses the issue of information loss in question and passage embeddings as the model depth increases, ensuring that critical details are retained throughout the reasoning process. On the other hand, multi-head attention proves particularly effective for complex numerical reasoning tasks, as it enables the extraction of multiple relevant information fragments and optimizes the solution process for multi-step problems.

RMNR, MANR, and RMMANR were evaluated on the FINQA dataset. Experimental results demonstrate that all three models outperform several baselines, with RMMANR achieving the highest performance by effectively combining the strengths of RMNR and MANR.

For future work, we plan to focus on enhancing the model’s performance on more complex multi-step reasoning tasks. This includes addressing challenges such as data scarcity for longer program steps and further optimizing the recall mechanism to reduce noise interference during operand prediction. Additionally, exploring alternative architectures or training strategies to improve generalization on complex reasoning tasks will be a key direction.