Enhancing Mathematical Knowledge Graphs with Large Language Models

Abstract

The rapid growth in scientific knowledge has created a critical need for advanced systems capable of managing mathematical knowledge at scale. This study presents a novel approach that integrates ontology-based knowledge representation with large language models (LLMs) to automate the extraction, organization, and reasoning of mathematical knowledge from LaTeX documents. The proposed system enhances Mathematical Knowledge Management (MKM) by enabling structured storage, semantic querying, and logical validation of mathematical statements. The key innovations include a lightweight ontology for modeling hypotheses, conclusions, and proofs, and algorithms for optimizing assumptions and generating pseudo-demonstrations. A user-friendly web interface supports visualization and interaction with the knowledge graph, facilitating tasks such as curriculum validation and intelligent tutoring. The results demonstrate high accuracy in mathematical statement extraction and ontology population, with potential scalability for handling large datasets. This work bridges the gap between symbolic knowledge and data-driven reasoning, offering a robust solution for scalable, interpretable, and precise MKM.

1. Introduction

Advancements in Artificial Intelligence (AI) have transformed numerous fields, including Mathematical Knowledge Management (MKM)—an interdisciplinary field concerned with representing, accessing, and maintaining mathematical knowledge using computational tools, encompassing areas like formal libraries, automated theorem proving, and mathematical search [1,2]. In this context, two key technologies stand out: knowledge graphs (KGs) and large language models (LLMs). KGs provide a structured and semantically rich representation of data, which is essential for managing complex mathematical knowledge. LLMs, in contrast, excel at understanding and generating natural language, enabling the extraction, interpretation, and organization of formal mathematical statements.

1.1. Context and Importance of the Study

The integration of KGs and LLMs has attracted considerable interest as it combines the advantages of structured and unstructured data processing. KGs facilitate advanced reasoning and semantic search by organizing data into meaningful relationships [3,4], while LLMs utilize extensive training on textual data to support a wide range of Natural Language Processing (NLP) tasks. Although both technologies have proven highly effective, their individual limitations become evident in scenarios that require high precision and domain-specific reasoning—such as those encountered in MKM. Recent empirical studies highlight significant failure modes: LLMs can exhibit high rates of hallucination (generating plausible but false information) and logical inconsistency, even when grounded with KG context [5,6].

By storing exact formulations, assumptions, and proofs within a KG, it becomes possible to rigorously support MKM.

1.2. State of the Research Field and Objectives

Recent studies emphasize the potential of combining structured ontologies with LLM-driven techniques to enhance domain-specific knowledge extraction and verification [7,8]. Nevertheless, challenges persist, particularly in the automated construction of high-quality KGs in domains where formal precision is critical [9,10].

Several influential approaches tackle the integration of KGs and LLMs for mathematical tasks, representing the current state of the art. Graph-Constrained Reasoning (GCR) frameworks aim to ensure faithful KG reasoning by integrating the KG structure directly into the LLM decoding process, thereby reducing hallucinations but often requiring complex KG curation [11]. Retrieval-Augmented Generation (RAG) pipelines, such as those evaluated by LemmaHead, focus on enhancing theorem proving by retrieving relevant context (definitions and lemmas) from authoritative sources, but performance still depends heavily on the quality and structure of the retrieved information [12]. Concurrently, initiatives like the Xena Project are building large-scale knowledge graphs based on formal proof assistant libraries like Lean 4 Mathlib, offering high logical rigor but demanding significant formalization effort [13,14]. Our approach contrasts with these by proposing a more lightweight and adaptable ontology designed for automated extraction from semi-structured LaTeX sources, aiming for a balance between scalability, formal structure, and ease of population, particularly suited for managing knowledge across diverse mathematical documents.

In MKM, ensuring the logical consistency and reproducibility of the underlying knowledge is essential. Our approach leverages LLMs to automate aspects of KG construction while employing a lightweight ontology-based framework to preserve logical integrity. This integration not only facilitates the validation of mathematical curricula but also enables the storage of precise computational methods, supporting their accurate retrieval and application in simulation workflows as needed.

1.3. Purpose, Aims, and Contributions

This study explores the integration of KGs and LLMs to address the key challenges in MKM. The central hypotheses of this work are as follows:

• The combined use of LLMs and ontology-driven KGs can significantly reduce the manual effort involved in constructing domain-specific knowledge repositories.
• Ontology-driven KGs improve the reasoning capabilities of LLMs by enforcing logical consistency and domain-specific constraints.
• The integrated approach provides a solution for validating mathematical curricula, demonstrating feasibility for handling large document sets through automated extraction and offering measurable consistency checks.

Our main contributions are as follows:

• A methodology that integrates KGs and LLMs for the extraction, validation, and management of mathematical knowledge.
• The design of a lightweight ontology to represent the structure of mathematical statements and proofs, thereby enabling advanced reasoning tasks.
• An automated extraction pipeline that processes structured LaTeX documents, allowing large-scale repositories—such as those from arXiv—to be efficiently incorporated into MKM systems.
• A quantitative evaluation on a synthetic mathematical dataset, demonstrating the effectiveness of the information extraction pipeline (Section 3), adopting evaluation principles aligned with rigorous benchmarking efforts like SBI-RAG for math word problems [15].

The remainder of this paper is structured as follows: Section 2 details the materials and methods, including the KG construction, ontology design, information extraction process, and use case algorithms. Section 3 presents the results of the information extraction evaluation and demonstrates the system’s use cases. Section 4 discusses the findings, technological challenges, limitations, and future directions. Finally, Section 5 concludes the paper by summarizing the key contributions and implications.

2. Materials and Methods

2.1. KG Construction and Maintenance

The construction and maintenance of KGs represent a fundamental challenge in the fields of AI and knowledge representation and reasoning (KRR). As the volume and complexity of data continue to increase, the need for robust and scalable solutions for KG management has become increasingly critical [4].

Recent research has explored various strategies to address the challenges associated with KG generation and upkeep. A key focus has been the development of techniques to ensure the consistency and validity of KG data over time. Hogan et al. [16] propose a framework for managing the evolution of KGs, emphasizing versioning, change detection, and update propagation. Their work underscores the importance of preserving the integrity of KG data as new information is added or existing data is revised.

In parallel, researchers have developed methods for automatically generating and populating KGs from heterogeneous data sources. Xue et al. [4] provide a comprehensive review of the state of the art in KG construction, highlighting the integration of structured and unstructured data and the application of machine learning techniques in this process.

The integration of LLMs with KG construction has emerged as a particularly promising area of research. Pan et al. [9] investigate the synergies between LLMs and KGs, demonstrating how their combination can enhance the accuracy, interpretability, and reasoning capabilities of AI systems. Their findings have inspired the development of novel approaches that leverage LLMs to support KG construction and maintenance.

One promising approach is the use of chain of thought (CoT) prompting, as described by Wei et al. [17]. CoT prompting guides LLMs to generate intermediate reasoning steps, thereby fostering a deeper understanding of the task. By incorporating such reasoning processes, CoT prompting improves the interaction between LLMs and task-specific objectives, such as KG generation. This technique leverages the user’s initial input to evoke more structured and relevant responses from the LLM, ultimately enhancing the accuracy and relevance of the generated KG data.

In addition, several researchers have investigated the integration of active learning and human-in-the-loop architectures to support KG construction and maintenance. Xue et al. [4] emphasize the value of incorporating user feedback and domain expertise to improve the accuracy and completeness of KGs over time.

2.2. Validation, Quality, and Reasoning

The use of ontologies and the SPARQL query language (https://www.w3.org/TR/sparql11-overview/ (accessed on 20 February 2025)) in our work provides a scalable approach to managing and querying the KG, offering efficient mechanisms for leveraging LLMs on large-scale KGs. The ontology-based representation enables efficient storage and retrieval of mathematical knowledge, while SPARQL queries support complex reasoning and inference over the graph.

The formal structure of the ontology—comprising defined classes, properties, and constraints—helps to ensure the consistency and validity of the represented mathematical content, serving as a quality control mechanism in KG construction. This structured approach not only enhances the reliability of the knowledge representation but also enables more accurate and meaningful interactions between the KG and LLMs, ultimately improving the overall system performance in mathematical reasoning and knowledge management.

Additionally, although our current implementation does not explicitly optimize for large-scale deployments, the architecture can inherently support scalability by clearly separating concerns between the LLM-serving layer and the KG-querying layer. For instance, it is possible to integrate an efficient serving infrastructure, such as deploying LLMs behind a scalable inference stack like vLLM, which manages GPU resources through pageable key–value caching [18]. Similarly, the KG querying could leverage distributed RDF databases, partitioning triples into shards managed by systems such as TriAD, which parallelizes SPARQL query execution across multiple nodes [19] and employs federation strategies inspired by FedX to minimize intermediate data transfer between shards [20]. Thus, the described architecture is inherently amenable to efficient distributed implementations, even though such optimizations remain beyond the scope of our current work.

2.3. Interfaces

The effective management and utilization of KGs require the development of user-friendly interfaces that support intuitive exploration, querying, and manipulation of the underlying data. As KGs grow in complexity and span diverse domains, the demand for accessible and interactive visualization tools has become increasingly important [21,22].

Previous research has emphasized the importance of designing interfaces that bridge the gap between the formal structured nature of KGs and the cognitive processes of human users [23,24]. Our work addresses several of these considerations by providing

• An interactive web-based interface for querying, visualizing, and editing the KG.
• Mechanisms for detecting and correcting errors in the extracted information, incorporating a basic human-in-the-loop approach.

While advanced features such as comprehensive KG versioning or active learning are beyond the scope of this study, the implemented web interface supports iterative improvements to data quality and lays the groundwork for future enhancements.

2.4. Information Extraction Process

The overall workflow of our information extraction system is illustrated in Figure 1. The system is designed to process LaTeX documents, extract blocks of mathematical statements (e.g., theorems, axioms, and definitions), identify relevant symbols or hypotheses, and incorporate them into a KG and ontology.

Below, we describe each stage of the pipeline in detail.

2.4.1. Stage 1: LaTeX Document Input Processing

The process begins with the input of one or more LaTeX documents. We apply

• A regular expression (RegEx) approach to identify structural elements, such as chapters, sections, and subsections.
• A hierarchical JSON representation is created to capture the document’s organization. For example:
  {
    “chapter1":   {
        “sectionA":   {
            “subsectionA1":   "text of that subsection"
          }
      }
  }

This step ensures a clear representation of the LaTeX document structure prior to the extraction of specific mathematical statements.

2.4.2. Stage 2: User Configuration

Users can configure the system by specifying the following:

• Which LaTeX blocks to extract (e.g., Theorem, Axiom, and Definition).
• Tags or labels to be assigned to specific chapters or sections for subsequent filtering.

This configuration is saved in a JSON file, such as

• {
    “hierarchy":   { …  },
    “latex":   {
       “theorem":   ["Theorem", "Corollary"]
      },
      “tags":   {
         “chapter1": ["Infinitesimal Calculus"]
      }
  }

2.4.3. Stage 3: Text Block Extraction

For each lowest-level subdivision (e.g., subsection or paragraph), the system applies

• A targeted RegEx to detect the user-specified blocks, such as
  \begin{Theorem}…\end{Theorem}.
• The extracted blocks are returned as an array of strings:
  ["\\begin{Theorem} … \\end{Theorem}",
   "\\begin{Axiom} … \\end{Axiom}", …]

This stage provides the foundation for subsequent analysis by isolating the mathematical blocks of interest.

2.4.4. Stage 4: Mathematical Formula Retrieval

Our approach to retrieving definitions and statements relies on the following components:

• Notational Assumptions: We assume consistent notation is maintained throughout the document—at least within the same mathematical domain—to ensure clarity and coherence.
• Semantic Representation: LaTeX expressions are transformed into semantic forms by mapping symbolic expressions to generalized tokens (e.g., set_1, set_2, element_1).
• Vector Representations: The preprocessed text is encoded into dense vectors using BERT-based models [25], including

  –

  AnReu/math_pretrained_bert [26].

  –

  math-similarity/Bert-MLM_arXiv-MP-class_zbMath [27].

  –

  MathBerta [28].

• Structure-Based Filtering: While symbolic layout trees or operator trees have been shown to improve retrieval precision [29], our approach prioritizes dense retrieval and does not rely on deep structural matching.

This method is particularly effective when LaTeX blocks contain short domain-specific statements, helping to ensure that retrieved definitions remain contextually relevant.

2.4.5. Stage 5: Retrieval from a Vector Database

Once the relevant text blocks are converted into vector embeddings, they are

• Indexed in a vector database, enabling efficient similarity-based searches.
• Utilized within a Retrieval-Augmented Generation (RAG) framework [30], in which the LLM is dynamically provided with the most relevant definitions or statements based on the user’s query.

2.4.6. Stage 6: Chain Process for Statement Analysis

We adopt a chain-of-thought approach [17] (chain) to analyze each extracted block:

• Detecting the Type of Statement: The system classifies each statement (e.g., theorem, axiom, and definition) based on the categories specified by the user in Stage 2.
• Symbol Extraction: A JSON-formatted list of symbols is generated, linking each MathematicalObject individual to its corresponding LaTeX representation. For example:
  [
    { "represents": "function", "represented": "f" },
    …
  ]
• Identifying Hypotheses and Conclusions: For statements with an if–then structure, the system explicitly separates the premise (hypothesis) from the consequence (conclusion).
• Creating Sub-Statements: In cases involving multiple clauses, the system decomposes compound implications into separate sub-statements, enabling more granular reasoning. This chain-based analysis also checks whether a statement was previously defined. If so, it is included among the retrieved statements from Stage 5; otherwise, it is classified under the UndefinedStatement class to maintain semantic rigor and traceability.

It is important to note that our current system primarily focuses on extracting explicitly stated knowledge and relationships present in the LaTeX source. The inference of implicit mathematical relationships (e.g., relationships not directly stated but logically derivable) is considered a direction for future work (see Section 4.4).

2.4.7. Stage 7: Final Ontology Verification

As the final step, newly extracted or updated statements are

• Verified against the ontology’s constraints to ensure consistency and validity.
• Stored in both the ontology and the vector database if they satisfy the consistency checks.

2.5. Front-End System Overview

A web-based user interface has been developed to streamline the workflows described above and to provide a comprehensive environment for MKM. Figure 2 presents the homepage of the interface, named Chatex.

The interface is organized into modular components, each tailored to a specific task in the MKM pipeline. Below, we describe the primary functionalities of these components.

2.5.1. Label Setup and Knowledge Base Population

Two main interface components support the extraction workflow:

• Label Setup: Users can configure labels for mathematical statements or document sections through a dedicated web interface. An example of this functionality is illustrated in Figure 3, where existing entries can be viewed, and new entries can be added or updated within the database.

  Figure 3. Chatex interface for managing labels in the database, including options to view existing entries and add or update new ones.

  Figure 3. Chatex interface for managing labels in the database, including options to view existing entries and add or update new ones.
• Populate Knowledge Base: Users can upload LaTeX files, specify which environments to track (e.g., Theorem and Definition), configure label assignments, and monitor the extraction process in real time. Figure 4 shows the interface for uploading documents to the knowledge base. Figure 5 presents the configuration page where users select elements to extract and apply labels. Finally, Figure 6 displays the processing logs generated during information extraction.

2.5.2. Similar Statements and Graph Visualizations

Once statements are extracted and indexed, users can query the system for similar statements, as illustrated in Figure 7. Retrieved results from the vector database are presented within an interactive interface, allowing rapid exploration and classification of analogous or related theorems, definitions, and lemmas.

Additional interface modules are available for advanced functionality, as illustrated in Figure 8. The example shown corresponds to the Maximize Conclusions module. These modules are presented here to demonstrate the interface design; the specific algorithms underlying each use case are detailed in Section 2.8.

2.5.3. Ontology Visualization and Export

After populating the knowledge base, the system offers graph visualization capabilities:

• Spring-Embedded and Shift Layouts: Spring-embedded layouts employ force-directed algorithms to emphasize clusters and reveal topological features. Shift layouts use geometric positioning to ensure planar crossing-free visualization of planar graphs.
• Hierarchical Layout: Based on the method of Sugiyama et al. [31], this layout organizes nodes into layers and reduces edge crossings, making it particularly suitable for visualizing logical inference flows.

An example of the ontology visualization interface is shown in Figure 9, displaying an RDF graph where predicates such as assumesThat and impliesThat can be selectively visualized using Spring Layout.

For advanced analysis, visualizations can be exported in GEXF format and imported into Gephi (https://gephi.org/ (accessed on 20 February 2025)), allowing for custom styling, complex filtering, and large-scale graph analytics.

2.6. On the Design of a Lightweight Ontology for MKM

A central contribution of this work is the development of a lightweight ontology, specifically designed to represent core elements of mathematical discourse in a machine-interpretable form. The ontology was implemented using the Web Ontology Language (OWL) and serves as the backbone of the system’s reasoning and validation capabilities. This section details its design objectives, class structure, logical foundations, and integration into the knowledge extraction workflow.

2.6.1. Scope, Purpose, and Intended Use

The ontology aims to formally model mathematical statements and their interrelations, including hypotheses, conclusions, definitions, proof steps, and mathematical objects. Its primary functions are

• Enabling the storage and retrieval of formalized knowledge.
• Supporting logical reasoning and validation of mathematical content.
• Serving as a semantic interface between LLM-generated content and symbolic knowledge.

2.6.2. Ontology Construction and Versioning

The ontology was developed iteratively, guided by the information extraction pipeline’s needs and the logical dependencies encountered in formal mathematical texts. The process included the following:

• Conceptual modeling: Identification of key conceptual categories in mathematical texts, such as Theorem, Definition, Axiom, Corollary, and ProofStep.
• Class definition: These concepts were formalized into OWL classes, organized under a root class MathematicalEntity. Core classes include
  • MathematicalStatement: superclass of Theorem, Definition, Axiom, Lemma, and Corollary.
  • MathematicalDescriptor: includes Symbol, Notation, and MathematicalObject (e.g., Function, Set, and Number).
  • MathematicalStep: includes reasoning methods such as Deduction, Induction, and ReductionToProblem.
• Property modeling: The following object properties were defined to capture logical structure:
  • assumesThat: links a statement to its hypotheses.
  • impliesThat: links a statement to its conclusions.
  • isProved / proves: bidirectional relation between statements and their proofs.
  • hasSymbol, represents, hasNotation: to connect symbolic representations with abstract entities.
• Logical constraints: Domain and range restrictions, disjointness axioms, and inverse property assertions were added to ensure semantic consistency.

A simplified view of the inferred class hierarchy is shown in Figure 10. The resulting ontology is organized around the following key classes:

• MathematicalStatement: Represents formal mathematical statements, including axioms, theorems, definitions, corollaries, and lemmas.
• MathematicalDescriptor: Encodes components used to describe mathematical content, such as notations, mathematical objects (e.g., functions, sets, and vectors), symbols, and proofs.
• MathematicalStep: Describes individual steps in a proof, including deduction, induction, recursion, and reductio ad absurdum.

The ontology defines several key properties to capture the logical relationships between mathematical statements and their components:

• assumesThat and impliesThat: These properties relate mathematical statements to their respective hypotheses and conclusions, enabling the representation of logical implication chains within the ontology.
• hasSymbol, hasNotation, and represents: These properties associate mathematical statements with the symbols, notations, and mathematical objects they reference, thereby supporting semantic linking and search.
• isProved and proves: These reciprocal properties establish the relationship between a mathematical statement and its proof. Specifically, isProved connects a statement to its corresponding proof, while proves asserts that a given proof validates a particular statement.

Consistency is ensured through both logical reasoners and manual verification.

Figure 10. Inferred hierarchy (i.e., the class structure derived automatically from logical axioms and subclass relations) of an ontology of mathematical results. The visualization shows classes hierarchically organized under owl:Thing, including main categories such as MathematicalStatement, MathematicalStep, object, and MathematicalDescriptor. It highlights different kinds of mathematical objects (e.g., set, function, and number), reasoning methods (deduction, induction, and ReductionToProblem), and foundational results (axiom, theorem, corollary, and definition). Is–a relations indicate conceptual specialization between entities.

Figure 10. Inferred hierarchy (i.e., the class structure derived automatically from logical axioms and subclass relations) of an ontology of mathematical results. The visualization shows classes hierarchically organized under owl:Thing, including main categories such as MathematicalStatement, MathematicalStep, object, and MathematicalDescriptor. It highlights different kinds of mathematical objects (e.g., set, function, and number), reasoning methods (deduction, induction, and ReductionToProblem), and foundational results (axiom, theorem, corollary, and definition). Is–a relations indicate conceptual specialization between entities.

2.6.3. Ontology Versioning and Availability

The current ontology version is 1.0 and is publicly available for reuse and extension. It can be accessed at

https://gitlab.com/universidad4774909/tfg/chatex-webui/-/raw/main/TFG1.owx

(accessed on 20 February 2025)

2.6.4. Integration with the Information Extraction Pipeline

The ontology is tightly integrated into the extraction pipeline and plays a role in each of the following stages:

• Post-processing: Each extracted block is matched against the ontology schema to identify its class.
• Semantic enrichment: Statements are linked to previously defined objects and notations.
• Reasoning: Ontology reasoners are used to infer class memberships and check for logical coherence. For example, if a new Theorem is linked to undefined Function symbols, these are flagged and stored under the UndefinedEntity class for later validation.
• SPARQL queries: These are used to detect implication chains, retrieve related definitions, and assess proof completeness.

2.6.5. Ontology Reasoning and Mathematical Information Extraction

The ontology is designed to work in conjunction with a logical reasoner, which ensures the consistency of the knowledge base and facilitates the inference of new relationships between mathematical entities. This reasoning capability enables

• Detection of logical inconsistencies or errors within stored statements.
• Deduction of implicit relationships and knowledge from the existing ontology structure.
• Validation of newly extracted statements by comparing them against established logical rules and semantic links.

To populate the ontology with mathematical knowledge, novel techniques were developed to extract information directly from LaTeX documents. Leveraging LLMs in combination with LangChain processing pipelines, the system can automatically identify and structure core components of mathematical discourse—including hypotheses, conclusions, and proof steps. This structured information is then integrated into the ontology, thereby enriching its content and enabling more sophisticated reasoning over the KG.

2.7. Prompt Templates

Throughout the information extraction process, we employ specialized prompt templates for LLMs to ensure consistency, accuracy, and relevance in the model outputs. Key considerations include

• Prompt Format: Different LLMs (e.g., llama3.1 or command-r) may require different prompt structures and formatting conventions [32].
• Few-Shot Examples: Where possible, prompts are augmented with annotated examples that demonstrate how to parse mathematical statements or identify hypotheses.
• Self-Reflection Approaches: Drawing on recent advances in reasoning strategies [33,34,35], prompts may include instructions that encourage the model to re-check or refine its intermediate outputs.
• Symbol Delimitation: Explicit guidance is provided to help the model distinguish between symbols and text, facilitating more robust semantic extraction.

By carefully designing these prompt templates and chaining multiple tasks (e.g., identifying statement type and extracting premises), we reduce inconsistencies and ensure high-quality representations of mathematical content in the KG and ontology. A representative prompt template is provided in Appendix A, and the complete set of carefully engineered templates tailored for Llama 3.1 is available at https://gitlab.com/universidad4774909/tfg/chatex-webui (accessed on 20 February 2025), facilitating reproducibility and further exploration.

2.8. Use Case Examples and Algorithmic Implementations

The integrated system supports a variety of use cases in MKM by combining graph-based algorithms with ontology-driven reasoning. This subsection outlines the principal algorithms implemented and their applications.

2.8.1. Optimization of Hypotheses

Objective

Given a set of hypotheses $H=\{h_1,h_2,\dots ,h_n\}$ and conclusions $C=\{c_1,c_2,\dots ,c_n\}$, all categorized under the definition class ($H,C\subseteq \mathit{Definition}$), the algorithm identifies the minimal set of additional hypotheses required to derive all conclusions in C.

Algorithmic Approach

Inspired by depth-first search (DFS), the algorithm recursively explores

• Which definitions or propositions in the ontology imply a given conclusion.
• The prerequisite assumptions needed for each implication to hold.

When multiple inference paths are available, the algorithm selects the one requiring the fewest additional hypotheses.

Output

The algorithm returns a mapping from each conclusion to the minimal set of required assumptions. For example:

• {
    “Proposition1”:   [
      “Definition_Anonymous”,
      “Definition_Compact_Interval”
     ],
    “Proposition3”:   [
        “Definition_Anonymous”
    ]
  }

Use Case

In an educational setting, this approach identifies the minimal prior knowledge needed to teach a specific concept. For instance, deriving the definition of a “bounded function” may require understanding of “rational functions,” “compact intervals,” and “denominator properties.” Similarly, the algorithm could analyze the minimal prerequisites for understanding Lebesgue integration, identifying that students should first be familiar with “measure theory,” “series convergence,” and “multivariable calculus,” following the structure found in second-year courses at the University of Seville [36].

2.8.2. Maximization of Conclusions

Objective

Given a set of hypotheses $H=\{h_1,h_2,\dots ,h_n\}$, the algorithm computes the maximal set of conclusions $C=\{c_1,c_2,\dots ,c_m\}$ derivable via the ontology’s impliesThat relationships.

Algorithmic Approach

Using a breadth-first search (BFS) strategy, the algorithm

• Initializes a queue with all hypotheses in H.
• Iteratively retrieves all conclusions implied by each hypothesis.
• Adds newly derived conclusions to the queue if they have not yet been processed.

Output

The result is the full set of reachable conclusions. Optionally, the derivation process can be visualized as a graph, with nodes representing statements and edges representing implication relationships.

Use Case

This algorithm assists with curriculum design by identifying the full range of concepts implied by a set of foundational topics. For example, starting with “rational functions” and “compact intervals,” it may derive “bounded functions” and related concepts, making explicit the implications of the selected curriculum base. Additionally, in a real educational scenario such as the third-year Algebraic Structures course at the University of Seville [36], instructors could input foundational topics like “Basic Algebra” and “Discrete Mathematics” to verify and visualize all higher-level algebraic concepts accessible to students after completion of these introductory courses.

2.8.3. Pseudo-Demonstrations

Objective

Given a set of hypotheses $H=\{h_1,h_2,\dots ,h_n\}$ and conclusions $C=\{c_1,c_2,\dots ,c_m\}$, the algorithm constructs a minimal hierarchical graph (pseudo-demonstration) that connects H to C via the shortest possible chain of impliesThat relationships.

Algorithmic Approach

A modified Dijkstra’s algorithm [37] is used as follows:

• Each implication edge is assigned a uniform weight of 1.
• The algorithm searches for the shortest paths from elements in H to each target conclusion in C.
• If a conclusion is unreachable, the system flags it as such.

Output

The output is a hierarchical graph that illustrates the minimal implication paths from H to C. Unreachable conclusions are clearly indicated.

Use Case

This tool supports logical reasoning and instructional design by visualizing the step-by-step progression needed to teach or derive a concept. For instance, it might show that the concept of a “bounded function” follows from prior knowledge of “rational functions,” “compact intervals,” and “denominator properties.” Similarly, in theorem-proving exercises, the pseudo-demonstration feature can help educators to visually demonstrate how certain propositions in Complex Analysis—such as Cauchy’s Integral Theorem—logically depend on previously established concepts like holomorphic functions and path integrals. This closely mirrors the pedagogical structure typically followed in Complex Analysis courses. The benefits are twofold: (1) students can verify whether they have already learned all prerequisite statements needed to understand a new concept, and (2) both students and instructors can easily design and evaluate a sequential study plan. In particular, this corresponds to a topological ordering as defined in [38], where each node (i.e., concept or statement) is only introduced after all its prerequisite nodes have been covered.

2.8.4. Academic Coherence and Course Planning

Objective

To support the design of academically coherent courses by analyzing the logical dependencies among mathematical concepts. Specific goals include

• Identifying concepts already covered in previous courses.
• Detecting missing prerequisites necessary to meet learning outcomes.
• Constructing pseudo-demonstrations that connect prior knowledge to new objectives.

Methodology

The course planning algorithm follows three main stages:

• Minimal Known Hypotheses: Filter out derivable content already covered in prior coursework to isolate the essential hypotheses.
• Hypothesis Minimization: Apply the Optimization of Hypotheses algorithm (Section 2.8.1) to determine the minimal set of additional prerequisites needed.
• Pseudo-demonstration Construction: Use the pseudo-demonstrations algorithm (Section 2.8.3) to build a hierarchical graph linking existing knowledge to new course content.

Use Case

Instructors can use this method to design courses that build systematically on students’ prior knowledge. By identifying gaps, reducing redundancy, and visually mapping dependencies, educators can ensure that course content follows a clear, logically grounded progression. For instance, this method can be practically applied to designing a coherent sequence of second- to third-year courses in the University of Seville mathematics program [36], clearly identifying and visually documenting how core second-year topics—such as “topology” and “Lebesgue integration”—serve as essential foundations for third-year courses like “Geometry and Topology of Surfaces.” For example, the Classification Theorem for Compact Surfaces depends on concepts such as the fundamental group and orientability, introduced in topology, while the Gauss–Bonnet Theorem requires measure-theoretic tools from Lebesgue integration. This approach makes it easy to analyze whether two courses can be taught simultaneously or if one must precede the other due to dependencies in their key statements.

2.9. Model Selection Criteria

Although the selection of LLMs is not the focus of this work, we identified certain capabilities as essential for the development of our framework: (1) mathematical reasoning, assessed using the Omni-math Benchmark [39]; and (2) faithfulness, evaluated via the Faith-Eval framework [40]. While our current implementation leverages these tools to inform LLM use, a comprehensive evaluation tailored to our specific application domain is needed for deployment in a production setting. Future work should include benchmarking these capabilities using domain-specific datasets, potentially integrating tools such as Ragas (https://docs.ragas.io/en/stable/ (accessed on 20 February 2025)) to assess reliability in retrieval-augmented generation pipelines.

It is important to note that our evaluation exclusively utilized open-source LLMs that were accessible and deployable on our local server infrastructure at the time the experiments were conducted.

3. Results

This section presents the evaluation of the proposed framework, focusing on two key aspects: (1) the quantitative effectiveness of the automated information extraction (IE) pipeline, and (2) a qualitative demonstration of the system’s utility through its core use cases.

3.1. Information Extraction Effectiveness

To rigorously assess the performance of the IE process, specifically the extraction of hypotheses and conclusions from mathematical statements, we developed a synthetic benchmark dataset. This dataset was generated using a Python script employing LLMs to create diverse mathematical statements across various fields, followed by manual correction and validation by the authors to ensure logical soundness and correctness. The generation script and resulting dataset are publicly available (https://gitlab.com/academic15/llm-synthetic-data-generation-mathematical-conlcusions-and-hypotheses (accessed on 20 February 2025)). An example entry from this dataset, illustrating the structured format, including definitions, hypotheses, conclusions, and field labels, is shown in Figure 11.

Following the guidelines outlined in [41,42], we designed a Proof of Concept (PoC) benchmark to evaluate IE performance. The benchmark adheres to the following principles:

• High performance on the benchmark should indicate robust in-domain task performance.
• Examples must be clearly annotated and unambiguous.
• Test samples should undergo thorough validation to eliminate erroneous or ambiguous cases.
• The dataset must provide sufficient statistical power for rigorous evaluation.
• The benchmark should identify and discourage the development of biased models by exposing potential harmful biases.

The dataset includes data evenly distributed across multiple branches of mathematics. It contains a diverse set of mathematical definitions, hypotheses, and conclusions from domains such as real analysis and group theory. Figure 11 shows a sample entry in JSON format, illustrating a theorem about finite cyclic groups within the domain of group theory.

Figure 12 presents the composition of the synthetic dataset, showing the number of samples distributed across different branches of mathematics.

The evaluation scripts and additional benchmarking tools are publicly available at https://gitlab.com/academic15/custom-benchmarking-llms (accessed on 20 February 2025).

First, we evaluated different embedding models for the task of retrieving relevant definitions, a crucial step in providing context to the LLMs (Stage 5 of the IE process, Section 2.4.4).

Table 1. Comparison of embedding models for definition retrieval.

Model	Accuracy	Recall	Execution Time (s)
Bert-MLM_arXiv-MP-class_zbMath	0.6281	0.6281	26.8666
math_pretrained_bert	0.6377	0.6377	28.3329
Bert-MLM_arXiv	0.6706	0.6706	29.6985

compares the performance of various embedding models in retrieving relevant definitions using the PoC dataset. The evaluation is based on accuracy and recall as the models were configured to return exactly 10 candidate definitions per query for use in downstream LLM prompts; thus, precision is not reported. The results indicate moderate performance, with math-specific pre-trained models like math_pretrained_bert [26] and Bert-MLM_arXiv [27] showing a slight advantage, underscoring the challenge of semantic retrieval in the mathematical domain.

Next, we evaluate and compare the performance of LLMs categorized by their size into three groups: small (2–3B parameters), medium (7–9B), and large (35–72B). This evaluation focuses on the core IE task: extracting hypotheses and conclusions based on provided context (definitions) and theorem statements. Performance was measured using standard metrics: accuracy, precision, recall, and F1 score, assessing the exact match of extracted hypothesis/conclusion sets against the ground truth.

3.1.1. Small LLMs

This category includes llama3.2:3B [43], phi3.5 [44], and gemma2:2B [45].

Table 2. Metrics for small models.

Model	Accuracy	Precision	Recall	F1 Score
gemma2:2B	0.7415	0.7311	0.9767	0.7415
llama3.2:3B	0.7281	0.7174	0.9651	0.7281
phi3.5:latest	0.7466	0.7436	0.9690	0.7466

summarizes their performance. Phi3.5 achieved the highest F1 score in this category, suggesting strong capability among smaller models for this structured extraction task.

3.1.2. Medium LLMs

This category includes mathstral [46], qwen2.5:7B, qwen2-math:7B [47,48], gemma2:9B [45], llama3:8B-instruct, and llama3.1:8B [49]. The results are presented in

Table 3. Metrics for medium models.

Model	Accuracy	Precision	Recall	F1 Score
mathstral:latest	0.7384	0.7308	0.9690	0.7384
qwen2.5:7B	0.7362	0.7291	0.9690	0.7362
qwen2-math:7B	0.7380	0.7238	0.9767	0.7380
gemma2:9B	0.7328	0.7257	0.9690	0.7328
llama3:8B-instruct	0.7336	0.7218	0.9690	0.7336
llama3.1:8B	0.7374	0.7234	0.9767	0.7374

. Those models specifically fine-tuned for mathematics (mathstral and qwen2-math:7B) generally performed well, with F1 scores around 0.74. This suggests that domain-specific training enhances performance on this structured extraction task, although the overall scores indicate that the task remains challenging.

3.1.3. Large LLMs

This category includes mixtral:8x7B-instruct-v0.1-fp16 [50], command-r:35B, qwen1.5:72B [51], llama2:70B-chat [52], llama3:70B-instruct, and llama3.1:70B [49].

Table 4. Metrics for large models.

Model	Accuracy	Precision	Recall	F1 Score
mixtral:8x7B-instruct-v0.1-fp16	0.7317	0.7176	0.9767	0.7317
command-r:35B	0.7352	0.7211	0.9767	0.7352
llama2:70B-chat	0.7319	0.7203	0.9690	0.7319
llama3:70B-instruct	0.7247	0.7145	0.9535	0.7480
llama3.1:70B	0.7258	0.7180	0.9535	0.7491
qwen1.5:72B	0.7402	0.7261	0.9767	0.7402

shows their performance. Interestingly, the largest models did not significantly outperform the medium or even the best small models on this specific task, with the F1 scores remaining around the 0.73–0.75 range. Qwen1.5:72B showed strong performance, comparable to the best small model. This suggests that, for structured extraction tasks like this, model scale beyond a certain point may yield diminishing returns compared to factors like fine-tuning or architectural choices. The achieved F1 scores, while respectable, indicate that errors still occur, likely due to the challenges in handling complex logical structures, parsing nuances in mathematical language, and potential LLM limitations like hallucination or reasoning gaps, as discussed in Section 4.1.1.

Figure 13 shows the mean execution time required by each large language model (LLM) to compute hypotheses and conclusions across the evaluation set. As expected, larger models incur significantly higher processing times, highlighting a crucial trade-off between model size, potential capability (although not always realized for this task), and practical deployment constraints.

Overall, the IE evaluation demonstrates the feasibility of using LLMs for extracting structured mathematical information, achieving F1 scores up to  0.75 on our benchmark. While this level of performance is promising for automating a KG population, it also indicates the need for robust validation mechanisms (like the ontology constraints and potential human-in-the-loop verification discussed earlier) to handle residual errors, especially given the high precision required in mathematical domains.

3.2. Use Case Demonstration

A complete walkthrough of the information extraction pipeline is available via a demonstration video: https://youtu.be/85kChrAtK1k (accessed on 20 February 2025). The use cases discussed in Section 2.8 are illustrated in an additional demo: https://youtu.be/tq-W4QAR1_s (accessed on 20 February 2025).

This demonstration highlights the following key steps:

• Label creation for filtering extracted statements.
• Ontology population from a well-structured LaTeX document.
• Visualization of the resulting KG using spring-embedded and planar layouts.
• Application of reasoning algorithms and visualization using the Sugiyama hierarchical layout.
• Ontology inspection and validation via the Protégé platform.

In summary, the use case demonstrations confirmed that the integrated system, combining LLM-based extraction with ontology-driven reasoning and visualization, provides a functional and potentially valuable tool for MKM tasks, such as curriculum analysis, knowledge exploration, and consistency checking.

4. Discussion

This work presents a comprehensive exploration of integrating LLMs with an ontology-based approach to construct KGs tailored for MKM. Throughout the project, several key challenges and opportunities were identified.

4.1. Technological Challenges and Limitations

This subsection outlines the primary technological challenges encountered during development, highlighting their implications and potential limitations.

4.1.1. Limitations of LLMs

Our experiments with diverse LLMs revealed several persistent limitations:

• Hallucinations: Fabrications of intermediate results, definitions, and proofs persisted despite meticulous prompt engineering. This behavior aligns with recent findings [5], which report pervasive hallucination across model sizes and architectures. Nevertheless, the integration of ontologies can impose domain-specific constraints, helping to delineate output boundaries in line with the Knowledge-Controlled Generation paradigm [53]. Moreover, the structured nature of the KG and subsequent validation steps offer potential mechanisms for identifying and mitigating some inconsistencies introduced by hallucinated content, although such challenges are not entirely eliminated.
• Symbolic Reasoning Deficiencies: Consistent with prior studies [10,54,55], the LLMs exhibited significant limitations in symbolic and multi-step mathematical reasoning. Their performance was often fragile, with minor semantically irrelevant changes (e.g., altering numerical values or introducing extraneous information) substantially impacting the outcomes, suggesting reliance on superficial pattern matching rather than robust logical inference [10]. Furthermore, the models frequently failed at compositional reasoning, struggling to spontaneously combine known concepts to solve novel problems involving logical traps [54]. Manual analyses confirm that these failures often stem from flawed logical chains, unwarranted assumptions, and difficulties regarding translating physical intuition into formalized steps [55]. Notably, although the expanded context windows permit the inclusion of additional relevant statements, doing so did not consistently enhance reasoning performance, underscoring a fundamental limitation in flexible knowledge integration rather than a mere context size constraint.
• Parameter and Language Constraints: Models with fewer than 5 billion parameters demonstrated limited capacity for parsing structured mathematical input and exhibited strong reliance on English-language prompts. These constraints reduced the performance robustness across diverse input scenarios. Nonetheless, recent advances in small language models suggest potential for future improvements [56].
• Dependency on Structured Input: Our system requires well-formatted LaTeX documents, limiting applicability to sources with clean standardized structures. Broader issues of unstructured document parsing, including neglect of tables, diagrams, and semantic metadata, remain challenging [57].

4.1.2. Challenges in Document Conversion and Standardization

Converting mathematical content from diverse sources, especially PDF documents, presented several difficulties:

• Conversion Quality: Many available tools and APIs generate cluttered LaTeX outputs that include unnecessary style elements, making it harder to identify structural components like theorems and definitions [58].
• OCR Limitations: Although advanced OCR models (e.g., [59]) were evaluated, they failed to provide the required precision and structure necessary for reliable extraction of complex mathematical notations.
• Heterogeneity of LaTeX Formats: The lack of a standardized LaTeX format significantly complicates automated extraction. The “ideal” LaTeX template we developed served as a controlled format to mitigate this issue, but widespread adoption of such a format remains a challenge.

4.1.3. Ontology Design and Prompt Engineering

A lightweight ontology, developed using OWL, was central to the structured representation of mathematical knowledge. It supports

• Enhanced consistency checking for newly extracted statements.
• Clear mapping of semantic relationships among theorems, definitions, and numerical methods.
• Integration with reasoning engines for validation, inference, and logical navigation through mathematical content.

In addition, prompt engineering played a crucial role in guiding LLM behavior. Techniques such as few-shot prompting, chain-of-thought reasoning, and structured symbol delimitation were used to mitigate errors and improve the semantic precision of extracted content [33,60].

4.1.4. Scope of Knowledge Extraction:

Our current approach primarily extracts explicitly stated knowledge and relies heavily on the information present within the source LaTeX documents. Systematically identifying potentially missing entities or relationships (e.g., concepts used but not formally defined within the processed text, and implicit logical dependencies) remains a significant challenge beyond simple label resolution. While parsing and validation steps catch some structural inconsistencies, comprehensive detection of semantic knowledge gaps is non-trivial. We currently assume that input documents largely follow a hierarchical dependency structure, aiding in the identification of some structural omissions, but acknowledge this limitation in capturing potentially implicit or missing knowledge, which is a key area for future work (see Section 4.4).

4.2. Cognitive Computing

The integration of symbolic (ontology-based) and subsymbolic (LLM-based) methods reflects a cognitive computing paradigm with clear implications for MKM. By emulating human cognitive processes—combining formal logical reasoning with intuitive language-based inference—this hybrid approach enhances

• The interpretability of complex mathematical models.
• The robustness and transparency of decision-making in knowledge-intensive environments.

4.3. Expanding the Methodology to Other Domains

Although this work focuses on MKM, the proposed methodology—integrating LLMs with ontology-guided KG construction—is adaptable to other domains.

In medicine, Arsenyan et al. [61] used LLMs to extract structured information from clinical notes, generating high-quality KGs for applications such as diagnostics and drug discovery. In law, Feng et al. [62] constructed KGs by aligning legal relations with domain ontologies and Wikidata, ensuring semantic consistency and interoperability.

In light of these findings, our methodology can be adapted to other domains by modifying three core components:

• Domain-specific ontology design: Replace the mathematical ontology with a structured vocabulary of entities and relations relevant to the target domain (e.g., biomedical conditions and legal concepts).
• Prompt specialization: Tailor prompt templates to reflect domain-relevant discourse structures and knowledge types using strategies like chain-of-thought prompting.
• Validation frameworks: Incorporate domain-appropriate quality checks, such as expert review for clinical safety or consistency audits for legal correctness.

These adaptations enable the construction of precise, interpretable, and domain-aware KGs suitable for high-stakes applications beyond MKM.

4.4. Future Directions

Several promising avenues for future research have been identified:

• Ontology Expansion: Expand the ontology to include mathematical and engineering concepts that support the direct execution of simulations or numerical methods when necessary.
• Improved Document Conversion: Develop AI-driven OCR pipelines, inspired by systems like DocLing [63], to reliably convert mathematical PDFs into structured LaTeX. This advancement would broaden the pipeline’s applicability to legacy and unstructured sources [57].
• Integration with Theorem Provers: Incorporate formal proof assistants (e.g., Lean and Coq) to validate and verify the correctness of extracted mathematical statements, strengthening the logical soundness of the knowledge base. Recent studies have shown impressive improvement regarding model performance when incorporating these tools into the LLM reasoning process [64,65].
• Advanced Prompt Engineering and Reasoning Model Integration: Develop and integrate prompt templates and reasoning strategies that leverage cutting-edge reasoning models [66] to perform multi-step internal planning to excel regarding advanced mathematics and science tasks and reduce LLM hallucinations [67]. This can enhance semantic fidelity, and extract richer symbolic representations.
• Bias Mitigation: Investigate and address the biases present in LLM outputs to ensure neutrality, reliability, and fairness in extracted content, particularly when deployed in educational or decision-support systems.
• Dynamic KG Maintenance: Addressing the dynamic nature of mathematical knowledge (e.g., revised definitions and evolving curricula) is crucial. Future work will explore a pipeline involving (1) versioned storage of triples with validity intervals; (2) change-triggered re-indexing based on fine-grained dependency tracking; (3) automated outdated-fact detection, potentially adapting techniques like Deep Outdated Fact Detection [68], which leverage structural and textual cues; and (4) minimal-change revision mechanisms, including archiving or consistent replacement of flagged triples to maintain KG integrity.
• KG Completion and Implicit Reasoning: A significant direction is enhancing KG completeness and inferring implicit knowledge beyond explicitly stated facts. Future work includes exploring text-enhanced KG completion methods. For instance, SimKGC [69] leverages contrastive learning on entity descriptions and is a natural baseline for our text-rich LaTeX sources. More recent neighborhood- and relation-aware models—KGC-ERC [70] and RAA-KGC [71]—set a new state of the art and should also be considered. To adapt these approaches to the mathematical domain, they should be combined with our mathematical-formula-retrieval pipeline (Section 2.4.4) so that symbols and equations provide additional domain-specific context for entities and relations.

5. Conclusions

This study has introduced and evaluated a novel framework integrating LLMs with an ontology-based approach for constructing and managing KGs specifically tailored for MKM. By automating the extraction, semantic organization, and logical verification of mathematical knowledge from structured LaTeX documents, our system offers a promising solution to the challenges of scalability, consistency, and precision in handling complex mathematical content.

5.1. Key Contributions

The primary contributions of this work can be summarized as follows:

• Ontology-Driven Knowledge Modeling: We developed a lightweight yet expressive ontology capable of representing core mathematical structures, including hypotheses, theorems, and proofs. This structured representation underpins the system’s ability to perform advanced reasoning, validate curriculum coherence, and retrieve relevant methods accurately [1,2].
• Automated Information Extraction Pipeline: A robust pipeline was implemented, leveraging state-of-the-art LLMs and vector retrieval models to accurately extract and structure formal statements from LaTeX source documents. This automation is key to enabling the ingestion and management of knowledge from large-scale repositories like arXiv, benefiting MKM [9,10].
• Hybrid Cognitive Computing Framework: The integration of probabilistic LLM-based generation with symbolic ontology-based reasoning embodies a hybrid AI approach. This synergy bridges the gap between data-driven pattern recognition and formal logic, resulting in a knowledge representation framework that is potentially more robust, explainable, and suitable for knowledge-intensive applications in education and research [72].

5.2. Implications for MKM

The practical implications of this framework are multifaceted, offering tangible benefits for MKM:

• Curriculum Validation and Model Verification: The system provides tools to ensure the logical flow and consistency of educational materials.
• Intelligent Tutoring Systems: By enabling the generation of pseudo-demonstrations and the verification of prerequisite knowledge, the framework can serve as a valuable component in developing sophisticated AI-driven educational platforms.
• Storage and Retrieval of Precise Methods: The KG acts as a repository for rigorously defined computational methods, such as precise algorithms implementing mathematical definitions or verifying properties, explicitly linked to the corresponding formal statements. This structured storage allows for the validation of computational implementations against their specifications, facilitates reliable retrieval, and promotes reproducibility and methodological rigor.

5.3. Acknowledged Limitations and Path Forward

While the results demonstrate the potential of our approach, realizing its full capabilities requires addressing certain limitations inherent in the current implementation and the underlying technologies. As detailed in the Discussion (Section 4.1.1), these include the system’s current dependence on well-structured LaTeX input, the persistent challenges associated with LLM performance in deep symbolic reasoning and potential for hallucination (despite mitigation strategies), and the necessarily bounded scope of the current ontology.

Our future research agenda, outlined in Section 4.4, is designed to directly address these points and enhance the system’s capabilities. The key directions include significantly expanding the ontology’s coverage into more specialized mathematical and engineering domains, developing more robust document conversion techniques (particularly for PDF sources), integrating formal verification tools like automated theorem provers to bolster logical soundness, and further refining LLM interaction strategies through advanced prompting and reasoning model integration to improve semantic fidelity and mitigate biases.

5.4. Final Remarks

In summary, this research demonstrates the significant potential of synergizing LLMs and ontology-driven KGs for advancing MKM. The focus on formal representation, automated and scalable information extraction, and logical coherence provides a powerful foundation for managing mathematical knowledge effectively, applicable in educational and research contexts. As symbolic and subsymbolic AI paradigms continue to evolve, this hybrid approach offers a promising trajectory for building more intelligent, interpretable, and domain-aware knowledge management systems.