A Formula-Structure-Embedding-Based GNN for Formula Equivalence Determination

Abstract

The complexity of mathematical formulas in scientific documents makes efficient formula similarity computation a critical task in information retrieval. Approaches based on text or structure matching cannot fully capture the inherent semantic and hierarchical properties of formulas. Although large language models (LLM) made significant progress in formula similarity comparison, the linear processing may ignore the structure information of formulas. In contrast, GNN excel at modeling the structure relationships of formulas, but still face challenges in accurately defining nodes and edges in a formula tree, especially in distinguishing between nodes at different levels and modeling edge relationships. In this paper, we propose a formula-structure-embedding based GNN (FSCMGNN), and it is integrated with formula structure embedding, structure standardization and attention mechanism. The model is designed for semantic equivalence comparison. Experimental results show that the model outperforms existing models in formula equivalence determination in calculus.

1. Introduction

Mathematical formulas, as structured information in scientific literature, constitute a fundamental part of scientific literature and serve as abstract representations of domain-specific knowledge. Different from texts, mathematical formulas have the advantage of accurately conveying complex mathematical meaning and logical relationships. Correct understanding of mathematical formulas not only helps to grasp the core content of scientific literature, but also is significant for knowledge extraction, automatic reasoning, and literature retrieval. Formula equivalence determination refers to the process of determining whether two or more formulas are equivalent under specific conditions, which facilitates researchers in overcoming superficial symbolic differences to semantically grasp the underlying meaning of formulas. It not only can help to improve the accuracy of mathematical information retrieval, but also can provide a solid foundation for intelligent processing tasks, e.g., mathematical text analysis and automatic reasoning, which has significant research value and potential for broad applications.

The methods for math formula equivalence determination can be divided into three categories: one is a category of methods based on symbolic reasoning [1,2], where a logic system or a computer algebra system is used to perform precise mathematical reasoning to determine formula equivalence. The methods of this category show a high degree of accuracy and interpretability, but have high requirements for professional mathematical knowledge and a low degree of automation. This makes it difficult to adapt to the needs of large-scale data processing. The second category of methods is mainly based on formula structure comparison [3,4]. These methods reduce manual intervention by defining structural similarities. However, explicitly representing and comparing all substructures demands significant computational resources, which limits their scalability and efficiency in practical applications. The third category of methods is based on machine learning [5]. These methods encode formulas into low-dimensional vectors via word embeddings or by pre-trained language models. Compared with the methods of two previous categories, methods of this category have the advantages of high computational efficiency and low computing resource requirements. However, their structure information is often lost during formula embedding.

In order to eliminate the effect of formula structure information losing in machine-learning-based methods, this paper proposes a multi-level node feature embedding based GNN (FSCMGNN) for formula equivalence determination. It integrates with formula standardization, formula node structure embedding and attention mechanisms to extract structure information effectively. Experiments show that this method achieves a significant performance improvement in the task of formula equivalence determination in calculus.

2. Related Work

Mathematical formulas are a special form of description. Early equivalence determinations are commonly used for information retrieval. Miner and Munavalli construct a mathematical n-gram-index to query similarities between mathematical expressions [6]. Yokoi and Aizawa improve the accuracy of formula similarity calculation by using a tree-structured formula representation method, in which they extract sets of sub-paths and computed their similarities [3]. Richard and Yuan use the vector space model with the TF-IDF algorithm and cosine similarity to assess the similarity between mathematical formulas [5]. Zanibbi and Blostein transform mathematical formulas into canonical sorted text strings with indexes, and the matching of sub-expressions is used as a judgement condition for similarity [7]. Kamali and Tompa propose a tree edit distance method that calculates formula structure similarity through node insertion and deletion operations [4].

The above methods define similarities between formulas, but they mainly focus on symbol-level similarity, lacking deeper semantic understanding. Therefore, they are not quite suitable for formula equivalence determination and mathematical automated reasoning. Zaremba et al. make initial progress in the search for mathematical identities. They propose TreeNNs to generate leaf node embeddings, which are recursively combined to obtain parent node vectors [8]. Lrving et al. unify character-level and word-level models and introduce a semantic embedding approach to enhance the generalization of automated reasoning [9]. Allamanis et al. propose a neural equivalence network based on TreeNN. This network is designed to learn semantic representations of successive algebraic and logical expressions. It can capture the semantic equivalence between expressions, even when they differ in grammatical form [10]. Davila and Zanibbi propose Symbol Layout Tree and Operator Tree for formulas representations. These representations provide options for subsequent formula tree parsing [11]. Gao et al. treat LaTeX formulas as linear text and develop Symbol2Vec based on the CBOW model to learn embeddings for LaTeX mathematical symbols [12]. Arashahi et al. apply Tree-LSTM to model the syntactic structure of differential equations, overcoming limitations in generality and scalability found in traditional numerical approaches [13]. Lample and Charton employ sequence-to-sequence models to solve symbolic mathematical tasks such as integration and ordinary differential equations [14]. Zhong et al. perform structural analysis of formulas by designing complex tree matching and pruning rules [15]. Peng et al. introduce MathBERT, a BERT-based model fine-tuned for mathematical formula understanding and applies it to multiple downstream tasks [16]. Li et al. leverage the Z3 theorem prover to assess symbolic equivalence, enabling identification of logically equivalent formulas with different symbols [17]. Raz et al. discover the intrinsic connection between formulas through matrix transformations. They propose that the equivalence of two mathematical formulas can be determined if their recurrence matrices satisfy a specific co-boundary condition. This approach effectively solves the equivalence determination problem related to mathematical constants [18].

3. Method

To address the limitations of existing methods in extracting structural features from mathematical formulas, this paper proposes a multi-level node feature embedding based GNN for formula equivalence determination. The model uses multi-level node feature embedding to capture structure features from different levels, providing a strong support for equivalence determination.

The model follows a typical encoder–processor–decoder architecture (see in Figure 1). The encoder consists of a formula standardization module, a multi-level node feature embedding module, a node-level embedding module that encodes each node’s representation, and a graph-level embedding module that captures the overall structure of the formula graph. The processor includes both graph-level and node-level similarity comparison modules, which determine formula equivalence from both global and local perspectives. Specifically, the graph-level similarity comparison module adopts the corresponding module of SimGNN to evaluate the overall structural similarity between formulas. The node-level similarity comparison module incorporates an attention mechanism that dynamically adjusts interaction weights between nodes, enabling more accurate determination of key structure components. Finally, in the decoder, the model employs a cross-attention mechanism to integrate similarity vectors from both levels and outputs the final equivalence prediction. The integration of formula standardization, multi-level node feature embedding, and attention mechanisms together enhances the model’s capability to effectively extract formula structure information and improve its scalability and accuracy in practical applications.

3.1. Encoder

The key components of the Encoder consist of a formula standardization module, a multi-level node feature embedding module, a node embedding module, a graph embedding module, and four sub-modules.

3.1.1. Formula Standardization Module

There are basic algebraic laws for formula transformation, e.g., associative law and commutative law, which make algebraic computation more flexible but increase the diversity of formula expressions, e.g., formulas in Figure 2 have the same mathematical meaning but different abstract syntax trees (ASTs). However, the diversity in form do not reflect any further information about domain knowledge. In order to eliminate the structure differences brought by using the basic algebraic laws, the formula standardization module is introduced to uniform the different abstract syntax trees caused by the use of these algebraic laws. In this module, the following typical operations are performed on each AST:

• When processing operators with associative property (such as addition or multiplication), this method examines whether the operators of the current node and its child nodes are the same: if yes, the associative law can be used on these two nodes, and other sub-trees of the current node and the two sub-trees of the child node will be set as the next children of the current node. This operation will be performed recursively operations until all operator nodes have been examined. After this step, the AST will become an N-ary tree (see in Figure 3).
• To process an operator with commutative property (such as addition or multiplication), we first provide a unique value for each type of operators or operands, then recursively calculate the dictionary order of each sub-tree, and rearrange the order of all sub-trees of the operator in ascending order of the sub-tree dictionary orders. The dictionary order is defined as a unique sequence obtained by recursively performing a breadth-first traversal, where the value of corresponding operator, the number of its sub-trees and the orders of all its sub-trees are sequentially recorded during the traversal process (see Figure 4).

By using the above operations, different forms of formulas caused by the application of algebraic laws can be standardized, which improves the accuracy of formula equivalence determination.

3.1.2. Multi-Level Node Feature Embedding Module

In mathematics, formulas can be represented as graphs, where nodes represent operators and operands, and edges represent operational relations between them. This graph preserves the hierarchical structure and logical relations of formulas. However, if we use the GNN model for formula equivalence determination directly, nodes are represented by their IDs. This representation fails to adequately use use the positional information of each node, resulting in an imprecise understanding of the formula’s overall structure. In order to solve this problem, we improve the representation of formulas based on the node structure information. In this paper, each operator or operand, such as ‘+’, ‘sin’, etc., is represented by a fixed number which is called the Node Type Identifier T); Node Level Information (L) refers the level at which the node is located in its formula tree; Node Intra-Level Number (N) records the relative position of the node at its level. The operations of formula structure embedding are as follows:

• Each node’s ‘Token T’, ‘Level L’, and ‘Number N’ are converted into corresponding vectors by embedding, such that each embedding vector contains all structure information of corresponding node.
• The three embedding vectors of a node are fused to form a node feature vector that contains all structure information of corresponding node (see Figure 5).
• A breadth-first traversal is performed on the formula tree to aggregate all node vectors, generating a vector that represents the structure information of the formula tree (see Figure 6).

Take ‘$x+2y+2xy$’ as an example,

Table 1. Characteristic representation of the formula.

ID	Symbol	Edges	Tokens	Levels	Numbers
0	+	(0, 1) (0, 2) (0, 3)	21	0	0
1	x	(1, 0)	50	1	0
2	×	(2, 0) (2, 4) (2, 5)	23	1	1
3	×	(3, 0) (3, 6) (3, 7) (3, 8)	23	1	2
4	2	(4, 2)	54	2	0
5	y	(5, 2)	51	2	1
6	2	(6, 3)	54	2	2
7	x	(7, 3)	50	2	3
8	y	(8, 3)	51	2	4

shows its structure information of each node. By encoding such structural information into node representations, the model incorporates hierarchical and positional details into computation, ensuring the accuracy and reliability of formula equivalence determination.

3.1.3. Node Embedding Module

In this paper, Neighbor-Aggregation-based Graph Convolution Network (GCN) [19] is used to capture the local structure of the graph. GCN performs convolution operations on each node by aggregating its own features with those of its neighboring nodes. As a result, each updated node representation encodes both the node’s original information and the structural context provided by its neighbors. The update process of node features in GCN can be represented as:

$$H^{(l+1)}=\sigma ({\widehat{D}}^{-{\displaystyle \frac{1}{2}}}\widehat{A}{D}^{-{\displaystyle \frac{1}{2}}}{H}^{\left(l\right)}{W}^{\left(l\right)})$$

(1)

Here, $H^{\left(1\right)}$ denotes the node feature matrix of layer l, $\widehat{A}=A+l$ is the adjacency matrix plus the self-loop, $\widehat{D}$ is the degree matrix of $\widehat{A}$, defined as ${\widehat{D}}_{ij}={\sum }_j{\widehat{A}}_{ij}$, $W^{\left(1\right)}$ is the weight matrix of the lth level, and $\sigma (·)$ is a nonlinear activation function, usually using ReLU. After multiple levels of GCN, the node embeddings are ready for inputing into the next stages.

3.1.4. Graph Embedding Module

In order to capture global features and combine node importance, a global context-aware attention mechanism is introduced to generate the global graph context vector by averaging and non-linearly transforming all node embedding vectors. Different attention weights W are assigned to the global context, and the global graph context vector calculated by Equation (2) reflects the overall structure information of the formula.

$$h=\sum _{n=1}^N\sigma \left(u_n^{T}c\right)u_n=\sum _{n=1}^N{f}_2\left(u_n^{T}tanh\left(\left({\displaystyle \frac{1}{N}}\sum _{m=1}^N{u}_m\right)W\right)\right)u_n$$

(2)

In Equation (2), N is the total number of nodes, $u_n\in R^D$ is an embedding of node n, $c\in R^D$ is a simple average of node embedding, $W\in R^{D\times D}$ is the learnable weight matrix, $\sigma (·)$ is the sigmoid activation function. This global context-based attention mechanism has an advantage of reflecting the overall structure features of formulas by flexible adjustment of node weights.

3.2. Processor

3.2.1. Graph–Graph Interaction Module

A simple way to model the relation between the graph-level embeddings is to calculate the inner product of the two. However, this often leads to insufficient interactions between graphs and makes it difficult to accurately measure the equivalence determination. To solve this problem, our method adopt the Neural Tensor Network (NTN) proposed by [20] to model the relation between two graph-level embeddings (see in Figure 7), i.e.,

$$g(h_i,h_j)=\sigma (h_i^T{W}^{\left[1,K\right]}{h}_j+V\left[{\displaystyle \frac{h_i}{h_j}}\right]+b)$$

(3)

Here, $h_i$ and $h_j$ are the graph-level vector representations of two formulas, $W^{\left[1:K\right]}$ is tensor parameters used to capture quadratic interactions between the vectors, V and b are weight matrices and bias terms for NTN, $\sigma (·)$ is a nonlinear activation function, and K is a hyperparameter that controls the number of interaction scores for each graph embedding pair.

3.2.2. Node–Node Interaction Module

Similar to the equivalence determination score in the graph–graph interaction module, we use a cross-attention mechanism instead of the inner product to calculate the node-level equivalence determination score (see in Figure 8). Unlike the inner product, which only captures the linear relationships between node feature vectors, the cross-attention mechanism introduces non-linear transformations that better capture complex interactions between node representations. The equations for the cross-attention mechanism are listed as follows:

$$Q_i=X_i{W}^{Q_i},Q_j=X_j{W}^{Q_j}$$

(4)

$$K_i=X_i{W}^{K_i},K_j=X_j{W}^{K_j}$$

(5)

$$V_i=X_i{W}^{V_i},V_j=X_j{W}^{V_j}$$

(6)

$$z_{i\to j}=AttentionScores=softmax\left({\displaystyle \frac{Q_i{K}_j^T}{\sqrt{d_k}}}\right)V_j$$

(7)

$$z_{j\to i}=AttentionScores=softmax\left({\displaystyle \frac{Q_j{K}_i^T}{\sqrt{d_k}}}\right)V_i$$

(8)

Here, z denotes the probability of formula equivalence, i and j denote the ith and jth nodes, $z_{i\to j}$ represents the attention that node i pays to node j, $z_{j\to i}$ indicates the attention from node j to node i, and X denotes the embedding of the input sequence, $W^Q$ and $W^K$ denote the weights learn from matrices Q and K, respectively.

3.3. Decoder

In the formula equivalence determination module, the graph level similarity score vectors and node level similarity score vectors are integrated by a cross-attention mechanism (see in Figure 9). The cross-attention mechanism can deeply interact with the two score vectors to capture the subtle relationship between them. The final score s for formula equivalence determination can be calculated as follows:

$$Q_G=g_G{W}^{Q_G},K_N=g_N{W}^{K_N},V_N=g_N{W}^{V_N}$$

(9)

$$s=softmax\left({\displaystyle \frac{Q_G{K}_N^T}{\sqrt{d_k}}}\right)V_N$$

(10)

where $Q_G$ is the Q matrix of the graph-level similarity vector $g_G$, $K_N$ is the K matrix of the node-level similarity vector $g_G$, and $V_N$ is the V matrix of the node-level similarity vector.

4. Experiments and Analysis

4.1. Dataset

To the best of our current knowledge, most of the existing research focus on the matching of formula symbols, and less on formula equivalence determination. Therefore, we construct a dataset from equivalent formulas in calculus, as calculus is not only an important part of higher mathematics and has a wide range of applications, but also involves a large number of equivalent transformations, i.e., derivative and integral transformation, Taylor expansion, trigonometric identity transformation, etc. These advantages allows us to easily obtain a large number of equivalent calculus formulas.

To obtain positive and negative samples for formula equivalence determination, a large number of mathematical expressions from several widely used calculus textbooks, i.e., Calculus and Its Applications [21] by Bittinger et al., Single Variable Calculus: Early Transcendentals [22] by Anton et al., and CALCULUS EARLY TRANSCENDENTALS [23] by Jon et al. are curated.

To generate positive samples, we applied four common mathematically equivalent transformations: algebraic expansions and simplifications, derivatives and integral calculations, substitution of variables, and trigonometric constant transformations to the extracted expressions, and used the Python’s SymPy tool (version 1.12) to generate equivalent formulas. These transformations were chosen because, firstly, algebraic expansion and simplification operations, e.g., ${(x+y)}^2=x^2+2xy+y^2$, are the common equivalence operations in elementary algebra, and are suitable for assessing the model’s understanding of the underlying transformations; secondly, derivative and integral calculations, e.g., ${\displaystyle \frac{d}{dx}}\left(x^2\right)=2x$, help to test the model’s grasp of the semantics of core operations in calculus; thirdly, variable substitution, e.g., $\int xdx={\displaystyle \frac{x^2}{2}}+C$ and $\int tdt={\displaystyle \frac{t^2}{2}}+C$, can verify whether the model has the ability to deal with the generalization of variable change without semantic change; lastly, the triangular constant transformations, e.g., ${sin}^2\left(x\right)+{cos}^2\left(x\right)=1$ and ${sin}^2\left(x\right)=1-{cos}^2\left(x\right)$, are highly structurally variable, which help to enhance the robustness of the model to complex structure deformations. These operations cover the most common cases of equivalent transformations in the field of calculus, reflecting both equivalence relations in the mathematical sense and structure diversity, which helps to train the model to maintain stable semantic comprehension under different structures.

To construct negative samples, we selected formulas with completely different mathematical meanings, e.g., $\int xdx={\displaystyle \frac{x^2}{2}}+C$ and $\int sin\left(x\right)dx=-cos\left(x\right)+C$, which may have structure similarities, but are unequivalent.

By the above method, a dataset of formula equivalence determination from calculus is constructed (see in

Table 2. Table of data set structure.

Books	Equivalent Pairs	Unequivalent Pairs
Single Variable Calculus: Early Transcendentals [22]	301	700
Calculus and its applications [21]	303	980
CALCULUS EARLY TRANSCENDENTALS [23]	459	1020
Jimidovich Mathematical Analysis Exercise Set [24]	590	1270

).

4.2. Evaluation Indicators

The metrics used in the experiments are $Accuracy$, $Precision$, $Recall$ and $F1$ score. Accuracy measures the proportion of samples correctly predicted by the model out of the total samples. Precision measures the proportion of correctly identified entities out of the identified entities. Recall evaluates the proportion of correctly identified entities out of all actual entities. F1 score is calculated as the reconciled mean of precision and recall as a combined indicator of model performance.

$$Accuracy={\displaystyle \frac{TP+TN}{FP+FN+TP+TN}}$$

(11)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(12)

$$Recall={\displaystyle \frac{TP}{FN+TP}}$$

(13)

$$F1={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$

(14)

where $TP$ denotes positive instances correctly predicted, $FP$ denotes negative instances misclassified as positive, $FN$ denotes positive instances misclassified as negative, and $FP$ denotes negative instances correctly predicted.

4.3. Experimental Results

To demonstrate the training process and convergence behavior of the proposed model, the training loss curve in Figure 10 is analyzed: the curve exhibits a consistent downward trend, indicating that the model is effectively minimizing the objective function; during the initial stages of training, the loss decreases rapidly, suggesting a quick learning from the training data by the model, while as the number of epochs increases, the rate of decrease in the loss slows down and gradually stabilizes, reflecting that the model parameters are approaching an optimal solution. This steady decline and plateau in the loss curve provide empirical evidence that the training process is both stable and effective, ensuring the reliability of the learned representations.

To demonstrate the superiority of our proposed model, comparative experiments among the pre-trained Bert model, the baseline model SimGNN, the GNN model NAGSL, and our proposed model FSCMGNN are conducted. As shown in Figure 11. The accuracy of the Bert model is 5.29%, 7.04%, and 13.31% lower than that of NAGSL, SimGNN, and FSCMGNN, respectively. This performance gap can be attributed to the nature of the models and the data: Bert, being optimized for natural language processing, excels at capturing semantic relationships in linearly structured text. However, when dealing with formulas represented as ASTs, its capabilities are limited. In contrast, GNNs are inherently better suited for such structure representations, as they operate directly on graph structures and can effectively aggregate both local and global structural information through message-passing mechanisms. These findings validate the suitability and effectiveness of GNN-based models for formula equivalence determination tasks.

To verify the effectiveness of each structural pre-processing module, we introduced Formula Structure Embedding (FSE) and Standardization (SD) operations in the Bert and SimGNN models, respectively, and conducted comparative experiments. The experimental results show that with only the introduction of formula structure embedding module, all indicators of SimGNN model have been improved, among which the F1 value has increased by 2.62%. Furthermore, after adding standardized operations on this basis, the F1 value increased by 2.1% again (see in

Table 3. Comparison of Bert and SimGNN with FSE and SD.

Model	Acc%	Pre%	Re%	F1%
Bert	50.75	46.02	56.53	50.74
Bert+FSE	69.5	60.11	76.72	67.41
Bert+FSE+SD	76.89	67.85	84.12	72.26
SimGNN	55.5	48.13	59.87	53.36
SimGNN+FSE	75.14	67.91	86.47	76.08
SimGNN+FSE+SD	83.93	71.02	90.41	79.56

). Similarly, the Bert model also shows some improvement after incorporating the same module, but overall performance still lags significantly behind the SimGNN model that introduced the same module. These results indicate that pre-processing structure information has a positive effect on improving model performance, especially in the graph neural network architecture. FSE and SD can more fully explore the structural and semantic features of formulas, further confirming the significant advantages of graph neural network-based models in mathematical formula equivalence determination.

To verify the effectiveness of our optimization modules, we conducted an ablation experiment based on the baseline model SimGNN, with results shown in Figure 12. Specifically, FSCMGNN integrates multi-level node feature embedding and several optimisation modules. To examine the impact of each component, we designed different variants: one without formula structure embedding (FSCMGNN-ff), one without the node-level similarity module (FSCMGNN-node), one without the similarity score calculation module (FSCMGNN-score), and one without standardisation processing (FSCMGNN-sd). As the results indicate, the formula structure embedding module contributes the most significantly to the overall performance, demonstrating that effectively incorporating structural information of formulas can greatly enhance similarity measurement. Standardization also plays a notable role by reducing redundant information and improving the consistency of feature representations, thereby facilitating better learning. Although the attention mechanism contributes relatively less to performance improvement, its ability to introduce non-linearity and enable the model to learn the relative importance of different features still brings observable effect.

5. Conclusions

In this paper, we propose a formula equivalence determination based on GNN. Specifically, the architecture of SimGNN combined with multi-level node feature embedding and standardization operation are used to effectively enhance the structure information extraction and processing capabilities of the model. In addition, by replacing linear operations with the attention mechanism for formula equivalence determination computation, the model’s ability to interact with global and local information is enhanced, thus improving the accuracy of similarity computation. In this paper, 5623 formulas in the field of calculus are extracted to form a dataset and validated on this dataset. The experimental results show that the proposed GNN-based method outperforms the baseline model in terms of accuracy and F1 value, showing that GNNs are more effective than large language models in handling mathematical formulas, which are highly structured data. In addition, by introducing structure information pre-processing, the accuracy of formula equivalence judgment is further improved, verifying both the advantage of GNN-based approaches and the importance of structure information enhancement for formula equivalence determination.

In future work, we plan to develop a context-combining retrieval framework that integrates both formula structure and its textual environment, aiming to support more accurate and semantically rich mathematical information retrieval.