Enhancing Geological Knowledge Engineering with Retrieval-Augmented Generation: A Case Study of the Qin–Hang Metallogenic Belt

Abstract

This study presents a domain-adapted retrieval-augmented generation (RAG) pipeline that integrates geological knowledge with large language models (LLMs) to support intelligent question answering in the metallogenic domain. Focusing on the Qin–Hang metallogenic belt in South China, we construct a bilingual question-answering (QA) corpus derived from 615 authoritative geological publications, covering topics such as regional tectonics, ore-forming processes, structural evolution, and mineral resources. Using the ChatGLM3-6B language model and LangChain framework, we embed the corpus into a semantic vector database via Sentence-BERT and FAISS, enabling dynamic retrieval and grounded response generation. The RAG-enhanced model significantly outperforms baseline LLMs—including ChatGPT-4, Bing, and Gemini—in a comparative evaluation using BLEU, precision, recall, and F1 metrics, achieving an F1 score of 0.8689. The approach demonstrates high domain adaptability and reproducibility. All datasets and codes are openly released to facilitate application in other metallogenic belts. This work illustrates the potential of LLM-based knowledge engineering to support digital geoscientific research and smart mining.

1. Introduction

With the rapid advancement of artificial intelligence, language models (LMs) have made significant strides in natural language processing (NLP), which are now capable of understanding, generating, and processing complex natural language texts through large-scale data training [1,2]. Recently, the continuous development of large language models (LLMs), such as GPT (Generative Pretrained Transformer) [3], BERT (Bidirectional Encoder Representations from Transformers) [4], Anthropic Claude [5], and LLaMA (Large Language Model Meta AI) [6], has driven rapid progress in NLP. Pretrained on extensive corpora, LLMs exhibit powerful general-purpose abilities across fields such as medicine [7,8], ecology [9], and civil engineering [10,11,12]. However, general-purpose LLMs, which rely on broad, nonspecific datasets, often lack the specialized knowledge required for complex tasks in specific fields [13,14,15,16]. As a result, these models may struggle with specialized questions, sometimes fabricating legal precedents [17], producing misinformation in news content [18], or even posing risks in clinical settings [19]. Without domain-specific knowledge, general LLMs frequently rely on templated responses, limiting their ability to deliver accurate and contextually relevant answers [20].

To overcome these challenges, Domain-Specific Large Language Models (DSLLMs) have emerged. Unlike general-purpose models (e.g., ChatGPT, Bing, Gemini [Google’s large language model], BERT), DSLLMs are designed and fine-tuned for specialized domains, incorporating domain-specific data to significantly enhance performance and reduce hallucinations (i.e., fabricated content) or irrelevant responses in complex tasks [21,22]. DSLLMs typically employ three approaches: prompt engineering, fine-tuning, and retrieval-augmented generation (RAG). Prompt engineering improves output quality by refining input prompts [23], and studies on prompt engineering for large models cover areas such as biomedicine, education, and academia [24,25,26]. For example, Heston et al. (2023) [27] utilized prompt engineering to optimize AI-driven generative language models (GLMs) in medical education; Lee et al. (2024) [28] developed an automated question generation (AQG) system through prompt engineering for use in English-language education; and Giray (2023) [29] examined prompt engineering’s role in enhancing ChatGPT for academic writing. In addition, Velásquez-Henao et al. (2023) [30] explored prompt engineering to improve interactions with language models in engineering applications.

Fine-tuning leverages labeled data from specific domains to further train large models, enhancing performance for domain-specific tasks [31]. Examples include BloombergGPT [32,33] and FinGPT [34] in the financial sector, as well as MedPaLM [35], HuatuoGPT [36], and BenTsao [37], which fine-tune Chinese LLMs for end-to-end medical consultations. In geosciences, models such as GeoGPT, SkyGPT, K2, and GeoGalactica specialize in geological tasks through fine-tuning. GeoGPT [38], for example, integrates geospatial data for diverse geospatial tasks, whereas SkyGPT [39] supports analysis and predictions in atmospheric science, climate change, and astronomy. Both K2 [40] and GeoGalactica [41] are trained on interdisciplinary datasets in geology, climate science, and remote sensing, catering to fields such as mathematical geology, biogeosciences, and geochemistry.

Although prompt engineering and fine-tuning enhance domain-specific performance, these methods can lead to “overfitting,” reducing effectiveness in broader tasks. To address this, retrieval-augmented generation (RAG) has been widely adopted. The RAG algorithm combines information retrieval with generation, allowing models to supplement generated content with external knowledge and reducing the likelihood of hallucinations [42]. Studies by Guu et al. (2020) [43], Gao et al. (2023) [44], and Fan et al. (2024) [45] demonstrate that combining dynamic retrieval with model pretraining addresses knowledge limitations and improves generation accuracy. Specifically, BloombergGPT uses RAG to access up-to-date financial information, enhancing accuracy and timeliness [46]. In medical dialogs, MedPaLM [47] applies RAG to improve responses to complex medical queries. RAG is also used in academic research, where SciBERT [48] leverages it to extract relevant literature content, generating research suggestions with improved relevance.

As a highly complex and data-intensive field, geology traditionally employs methods such as field surveys, geological mapping [49,50,51], numerical simulation [52,53], geophysical exploration [54,55], and remote sensing technology [56,57]. These methods provide the foundation for understanding geological phenomena. However, with the rapid growth of data volumes and the emergence of heterogeneous multisource data, traditional approaches struggle to address challenges related to multiscale problems and data integration [58]. Knowledge graphs, which offer a semantically structured approach for storing and reasoning with information, show significant application potential in this context. For example, Zhou et al. (2021) [59] constructed a knowledge graph for porphyry copper deposits, linking “earth systems, mineralization systems, exploration systems, and predictive evaluation systems,” to offer an intelligent solution for mineral resource forecasting. Zhang et al. (2024) [60] developed a knowledge graph focused on porphyry copper prospecting in the Qin–Hang metallogenic belt, integrating natural language processing and machine learning for automated knowledge extraction, reasoning, and visualization. This system provides efficient, accurate support for mineral prospecting. Zhang et al. (2024) [61] further developed an ontology library for mineral deposits via Neo4j, highlighting ontology’s role in data structuring and inferential analysis and addressing the challenges of semantic description and interpretability for multisource data.

Recently, intelligent dialog systems have introduced new tools for geological research by rapidly integrating and retrieving multisource data. Key question-answering systems in this field include GeoDeepDive [62], GeoAI [63,64], and SeisBench [65]. For example, GeoDeepDive [62] combines natural language processing (NLP) and machine learning to focus on extracting information from geological literature, allowing researchers to locate information more efficiently. This system is widely used across subfields; for example, Maio et al. [66] employed GeoDeepDive for entity linking and uncertainty management in geological research, whereas Goring et al. [67] used GeoDeepDive workflows to identify Ice-Rafted Debris (IRD) from the Pleistocene and Pliocene epochs. Additionally, Husson et al. [68] and Kumpf et al. [69] integrated GeoDeepDive with macrostrat, advancing deep-time research and accelerating the identification of geological evidence for evaporites. GeoAI [70], on the other hand, specializes in geological exploration and data analysis, combines GIS and deep learning models to address questions related to geological features and resources, playing a significant role in urban planning [70] and environmental protection [71]. SeisBench [72], an open-source deep learning tool for seismic data, integrates seismological data with NLP technologies for classifying seismic waves and event identification, supporting applications such as earthquake event detection [73,74] and source parameter estimation [62,75]. Despite their strong performance in specific areas, these models often lack broad interoperability, real-time processing capabilities, and data integration. For example, GeoDeepDive is focused primarily on geological literature extraction, whereas GeoAI specializes in geological exploration and data analysis; both tend to work with static historical data and lack real-time analysis capabilities. SeisBench also faces challenges in processing large-scale, real-time data streams, especially in responding quickly to earthquake events. Through multimodal data integration, cross-domain processing, and enhanced generalization capabilities, domain-specific large models offer a promising approach to address these limitations, providing more flexible and comprehensive support for geological research [76].

To address these challenges, this study leverages the ChatGLM3-6B model, combined with the open-source Langchain framework, to develop a domain-specific large language model tailored for geology, using the Qin–Hang metallogenic belt as a case study. By integrating literature and database resources, this study enables efficient knowledge retrieval and automated processing of heterogeneous data, building an intelligent question-answering system that encompasses multidisciplinary knowledge and enhances automated responses to questions related to the Qin–Hang metallogenic belt. The study involves the following steps: (1) Model and application development on the basis of the ChatGLM3-6B model and the Langchain framework; (2) construction of a comprehensive knowledge base covering the Qin–Hang metallogenic belt, including academic papers, geological reports, and book materials; and (3) evaluation and testing to optimize the performance of the intelligent question-answering system. This study provides a new tool for research on the Qin–Hang metallogenic belt, supporting the intelligent advancement of geological research in this region.

2. Related Research

2.1. Geological Background and Research Significance

The Qin–Hang metallogenic belt, which is part of the South China metallogenic province, represents a plate junction zone formed by the Neoproterozoic collision of the Yangtze and Cathaysia ancient continental blocks. During the Mesozoic, it became the origin of the Yanshanian tectonic movement in South China, establishing itself as one of southeastern China’s most critical magmatic metallogenic belts [77,78]. This NE-trending belt spans approximately 2000 km from Qinzhou Bay in Guangxi through eastern Hunan and central Jiangxi to Hangzhou Bay in Zhejiang, with a width of 100–300 km and an overall arc-like shape [60]. Recent studies have recognized the Qin–Hang belt as a major and distinct polymetallic metallogenic zone in South China, featuring widely distributed copper, lead-zinc, tin, and tungsten deposits [79,80]. Geologically, it plays a pivotal role in South China’s tectonic framework, with unique characteristics that illuminate the region’s tectonic evolution. Analyzing its geological features and evolutionary history can support the assessment and prediction of potential geological hazards, such as earthquakes and landslides, providing a scientific basis for disaster risk mitigation [81]. The formation of this belt reflects the complexity of crustal movements and sedimentary environments and contributes valuable insights into the global Grenvillian orogeny, Rodinia supercontinent breakup, and Neoproterozoic metallogenic activities [79,82,83,84,85,86,87,88].

In geological exploration and research, however, challenges arise from scattered information, data heterogeneity, and the difficulty of integrating interdisciplinary knowledge. Data and findings related to the Qin–Hang belt are spread across various publications, databases, and exploration reports, making traditional retrieval methods inefficient and prone to omissions. Additionally, geological exploration involves diverse data types—such as geological maps, mineral distribution data, exploration records, and chemical analyses—which require significant manual processing and format conversion due to inconsistencies. Moreover, research on the Qin–Hang belt spans several disciplines, including mineralogy, geochemistry, and structural geology, making interdisciplinary data integration a complex, time-intensive task that hinders scientific progress.

As a result, developing a domain-specific large language model (DSLLM) for the Qin–Hang metallogenic belt has both academic and practical importance. By integrating extensive heterogeneous geological data and literature resources, such a model could significantly improve data processing efficiency and knowledge acquisition, streamlining traditional geological exploration workflows. This intelligent support can provide precise and efficient mineral resource assessments, drive further research on the Qin–Hang belt, and offer a scientific foundation for regional resource development.

To facilitate a better understanding of the spatial and structural characteristics of the Qin–Hang metallogenic belt, a simplified geological map illustrating the regional tectonic framework, major faults, and Yanshanian magmatic intrusions is provided in Figure 1.

2.2. Research on the ChatGLM3-6B Model

This study utilizes the ChatGLM3-6B model as its foundational framework. Developed by Tsinghua University’s Zhipu AI team and the Beijing Academy of Artificial Intelligence, ChatGLM3-6B is a third-generation bilingual pretrained model within the GLM series featuring 6 billion parameters. Despite its relatively modest parameter size, ChatGLM3-6B exhibits strong capabilities in understanding and generating Chinese text, as well as handling English, owing to its optimized architecture. These characteristics enable it to excel in cross-lingual tasks and manage complex scenarios across multiple languages [89].

Compared with its predecessors, ChatGLM3-6B retains advantages in conversational fluency, fast response, and ease of deployment while further enhancing performance, particularly in environments with limited computing resources. Its compact and efficient design allows deployment both in the cloud and on local devices, making it suitable for a broad range of applications, from personal inference tasks to enterprise-level localized services. ChatGLM3-6B’s optimizations extend to handle complex interactions, including task-oriented dialog, open-domain question answering, and multiturn conversation scenarios. In practical dialog applications, the model’s robust context memory enables coherent multiturn exchanges. Across eight benchmark Chinese and English datasets, ChatGLM3-6B has demonstrated significant improvements over earlier versions, showcasing adaptability and robustness in bilingual settings. Its performance meets both academic and practical requirements for efficiency and reliability, making it widely applicable in intelligent dialog, knowledge-based question answering, and text generation.

2.3. Overview of the LangChain Open-Source Framework

LangChain is an open-source framework designed to facilitate the development of applications based on large language models (LLMs). Key features include chain-based task execution, intelligent agents, memory mechanisms, prompt templates, and integration with external tools. These features work together to enable developers to rapidly build complex automated systems and versatile applications such as intelligent dialog systems, Q&A platforms, and information retrieval tools. As shown in Figure 2, the LangChain framework is structured into five layers: observability, deployments, cognitive architectures, integration components, and protocols. Each layer serves a distinct role in simplifying and supporting the development, integration, and deployment of LLM applications [90]. A detailed overview of each layer follows:

• Observability Layer: LangSmith at the top level offers tools to monitor and optimize application performance in real time. Debugging allows developers to debug applications during development to quickly identify and resolve issues. Playground: An interactive testing platform for verifying application performance and behavior in real-world conditions. Evaluation: This tool provides tools for performance and accuracy assessment, helping improve response quality. Annotation: Facilitates data annotation and labeling for effective data management during testing and training. Monitoring: Offers real-time application monitoring to identify potential issues and support preventive maintenance. The observability layer provides LangChain with comprehensive debugging and monitoring capabilities, allowing developers to test, analyze, and improve application performance effectively.
• Deployments Layer: This includes LangServe and Templates, which focus on deployment and templates for applications. LangServe enables the deployment of chain-based tasks as REST APIs, allowing LangChain applications to integrate seamlessly into broader systems. Templates: Preconfigured application templates to help developers quickly build and configure applications for common tasks. The deployment layer simplifies application integration and expansion, making it easier for developers to publish applications as REST APIs or quickly setting up specific applications via templates.
• Cognitive Architectures Layer: Comprising LangChain’s core modules, this layer is essential for building intelligent applications, including the following: Chains: Supports chain-based task execution, allowing developers to create workflows by linking operational steps. Agents: Intelligent task units that dynamically make decisions, choose tools or execute specific tasks on the basis of requirements. Retrieval strategies: These strategies offer various methods for extracting relevant information from large datasets, such as in knowledge-based Q&A. These cognitive architectures provide developers with tools to build complex and adaptable applications capable of automated decision-making and multistep tasks.
• Integration Components Layer: Contains the LangChain-Community module, which integrates components for model interaction and retrieval, providing Model I/O, which includes components for model calls, prompt templates, example selectors, and output parsing. Retrieval: This method supports functions such as document loading, vector storage, and embedding models for precise information retrieval. Agent Tooling: Offers toolkits to enhance agent interactivity and data processing within applications. These components enable rapid setup and customization for applications that interact with LLMs, increasing flexibility and scalability.
• Protocol Layer: The LangChain Expression Language (LCE-L) in this layer enables essential features such as parallelization, fault handling, and asynchronous execution, providing low-level support that ensures high efficiency and scalability across other modules.

2.4. Overview of LangChain-Chatchat

LangChain-Chatchat is a local knowledge base Q&A application built on the LangChain framework and designed for offline use in Chinese scenarios. Leveraging the LangChain framework and large language models (such as ChatGLM), LangChain-Chatchat enables efficient Chinese and English information retrieval and Q&A. By supporting custom local knowledge bases, LangChain-Chatchat allows users to perform Q&A without relying on online services. Its design is well suited to the Chinese context, handling diverse input requirements flexibly for general inquiries and specialized queries alike. Additionally, the project supports multiple open-source LLMs and embedding models, allowing it to be fully built on an open-source tech stack for private deployment with high control [91].

LangChain-Chatchat operates entirely offline, ensuring data privacy by enabling local deployment, which is especially useful for sensitive scenarios within internal corporate or organizational networks. Without relying on cloud services, LangChain-Chatchat completes data processing and storage locally, making it ideal for environments with strict privacy and security needs. With embedding models, LangChain-Chatchat converts knowledge base text into vector representations, enabling vector retrieval to quickly find answers most relevant to user queries. Large language models such as ChatGLM then generate coherent and natural responses. Users can import various document types (PDF, Word, Excel, etc.) into the knowledge base, creating rich local Q&A resources. By combining LangChain’s flexibility with ChatGLM’s generative capabilities, LangChain-Chatchat offers an efficient, secure, and easily deployable knowledge-based Q&A solution for local use.

3. Knowledge Base Construction

3.1. Technical Approach

Retrieval-augmented generation (RAG) technology combines information retrieval with generative language models, enabling dialog systems to dynamically access external knowledge to generate more accurate and contextually relevant responses. Leveraging the LangChain framework, RAG facilitates the development of efficient and reliable Q&A systems by loading and processing local knowledge bases. The key steps are as follows (Figure 3) [92]:

• Data Loading and Preprocessing: Various file types within the local knowledge base (e.g., HTML, MD, JSON, CSV, PDF, and TXT) are parsed via an OCR parser to extract content, converting it to a standard text format for further processing. An unstructured data loader ensures compatibility with subsequent processing steps. The extracted text is then segmented on the basis of predefined content and length parameters via the RecursiveCharacterTextSplitter tokenizer, maintaining logical coherence at the sentence or paragraph level.
• Text Vectorization: Each text segment is converted into a vector representation via the bge-large-zh-v1.5 embedding model to capture its semantic features. This process encodes each segment as a high-dimensional vector, which is stored in a vector database (Faiss) to facilitate similarity-based retrieval. Text vectorization not only retains semantic content but also enables efficient similarity searches through vector operations.
• Vector Indexing for Retrieval: All text vectors are stored in the Faiss database to construct a vector index, enabling approximate nearest neighbor (ANN) searches. When a user submits a question, the system can quickly identify similar text segments within the vector space. To optimize the retrieval speed, the system chooses an appropriate indexing structure (e.g., HNSW or IVF) on the basis of the size of the knowledge base and uses cases, allowing rapid, accurate retrieval of relevant information from large datasets.
• Query Vectorization and Retrieval: Upon receiving a user query, the system first vectorizes the question. It then matches this query vector with stored vectors in the database to retrieve the top-K most relevant text segments. These retrieved segments serve as context, forming a “question + context” prompt that is fed into the generative language model to enhance response relevance.
• Answer Generation: The ChatGLM3-6B generative model processes the “question + context” prompt, synthesizing a response that draws on both the model’s intrinsic knowledge and the retrieved content. By accessing up-to-date external information from the knowledge base, the RAG approach enhances response accuracy and relevance, allowing the model to generate answers that are better tailored to complex user queries.

3.2. Data Collection and Preprocessing

A total of 615 research papers on the Qin–Hang metallogenic belt were collected from various academic platforms and publications. Specifically, the China National Knowledge Infrastructure (CNKI) database includes 213 papers, Google Scholar hosts 240 papers, the VIP database provides 7 papers, MDPI contains 27 papers, Elsevier journals feature 118 papers, and Springer journals offer 5 papers. Additionally, 5 specialized books cover research in this area.

4. Analysis and Evaluation of Intelligent Dialog Results

4.1. Dialog Interface

As shown in Figure 4, on the dialog interface, when a researcher asks, “What are the main geological factors affecting the mineralization of the Qin–Hang metallogenic belt?”, the model processes the query by first vectorizing it with RAG technology and retrieving the most relevant content segments from the knowledge base. These retrieved segments serve as contextual information that, combined with the user’s question, is fed into the generative model to generate a response to the geological factors influencing mineralization in the Qin–Hang belt. Once the response is generated, the model also displays the background literature and research reports it used, providing researchers with insight into the sources and rationale behind the answer. This approach not only enhances the response’s credibility but also offers researchers additional reference materials for a more thorough understanding of the geological features and mineralization processes of the Qin–Hang belt. This transparency in information improves the user experience and supports further research and learning in geology.

As shown in Figure 5, when the knowledge base is not utilized, the base model, ChatGLM3-6B, directly generates the response. In this case, the model relies solely on general pretrained knowledge without drawing on specialized geological information. While the base model provides broad knowledge coverage, it often lacks the specificity and depth required for specialized fields such as the Qin–Hang metallogenic belt. In contrast, integrating RAG technology with a knowledge base retrieval mechanism significantly improves the accuracy and professionalism of responses, as well as their credibility. This comparison highlights that combining RAG technology with a specialized knowledge base is essential for generating high-quality, expert-level answers, providing researchers with valuable reference information.

4.2. Comparative Analysis of Dialog Responses

The ChatGLM3-6B RAG-based intelligent conversational knowledge base clearly demonstrates an advantage in professional expertise when generating responses related to the Qin–Hang metallogenic belt, as shown in Figure 3. It provides specialized terminology and detailed perspectives on the belt’s structural characteristics, rock types, and mineralization conditions. For example, it highlights the significance of the uplifted and folded zones within the belt, which are critical structural features linked to favorable geological conditions for mineralization. Additionally, ChatGLM3-6B lists the main rock types essential for understanding the belt’s foundational geology, offering a focused approach that presents concise, relevant information for users with some background knowledge in the field.

ChatCPT-4o [93], as shown in

Table 1. Answers generated by ChatCPT- and Bing.

Model	Answer
ChatCPT-4o
Bing

, also performs well in providing a clear and organized explanation of the tectonic environment, magmatic activity, and hydrothermal processes. It provides comprehensive coverage of geological factors, especially in explaining how structural intersections affect mineralizing fluids, displaying a logical and coherent structure. ChatCPT-4o’s clarity and logical flow make it particularly suitable for users seeking a broad understanding of mineralization without necessarily having deep domain knowledge.

By comparison, Bing [94] covers essential geological factors but remains relatively introductory from a professional perspective, as shown in Table 1. While it introduces geophysical elements, such as gravity and magnetic anomalies, which add a unique perspective, its content lacks the depth seen in ChatGLM3-6B and ChatCPT-4o, particularly in the detailed explanation of key geological processes and mineralization mechanisms. This makes Bing’s response more suited as an introductory overview, potentially lacking the depth required by professional users.

In summary, the ChatGLM3-6B RAG excels in delivering specialized, domain-specific information, making it an effective knowledge support tool for the Qin–Hang metallogenic belt. ChatCPT- performs well in terms of clarity and logical structure, making it ideal for users looking for a comprehensive, general overview. Bing provides a basic overview in this domain, covering essential points but with limited depth.

4.3. Evaluation

4.3.1. Introduction to the BLEU Evaluation Method

This study uses the BLEU (bilingual evaluation under study) scoring method [95] to compare the Qin–Hang metallogenic belt knowledge content generated by five large language models (ChatGLM3-6B, ChatGLM3-6B RAG, ChatCPT-4o [93], Bing [94], Gemini [96]). BLEU is an automated evaluation metric that is based on n-gram matching and is used to measure the similarity between generated and reference texts. BLEU scoring includes the following key steps:

n-gram Matching: The BLEU segments both the generated and reference texts into n-grams and counts the occurrences of each n-gram in both texts, taking the minimum count.

n-gram Precision Calculation: The BLEU calculates precision for 1- to 4-g matches via the formula for n-gram precision PnP_nPn shown in Equation (1).

Brevity penalty (BP): To prevent excessively short responses, the BLEU applies a brevity penalty (BP) if the generated text is significantly shorter than the reference text, which is calculated as shown in Equation (2).

BLEU Score Calculation: Combining n-gram precision and the brevity penalty, the BLEU score is calculated as shown in Equation (3).

$$P_n={\displaystyle \frac{{\sum }_{all n-grams}min({count}_{gen},{count}_{ref})}{{\sum }_{n-gram}{count}_{gen}}}$$

(1)

$$BP=\left\{\begin{array}{ll}1& if c>r\\ e^{(1-r/c)}& if c\le r\end{array}\right.$$

(2)

$$BLEU=BP\times exp({\displaystyle \frac{1}{N}}\sum _{n=1}^{N}log{P}_n)$$

(3)

where

P_n: Precision for n-grams;

c: Length of the generated text;

r: Length of the reference text;

BP: Brevity penalty;

N: Maximum n-gram order, typically set to 4;

count_gen: Count of n-grams in the generated text;

count_ref: Count of n-grams in the reference text;

4.3.2. Evaluation Metrics

In addition to the BLEU, this study uses the precision, recall, and F1 score to evaluate the quality of model-generated content in the Qin–Hang metallogenic belt. Each metric provides insight into different aspects of generation quality:

The precision (Equation (4)) represents the proportion of n-grams in the generated text that match the reference text, indicating accuracy.

Recall (Equation (5)) measures the proportion of reference text content that is covered by the generated text, reflecting comprehensiveness.

The F1 score (Equation (6)) balances precision and recall, providing a comprehensive assessment of overall performance.

$$Precision={\displaystyle \frac{M}{N_g}}$$

(4)

$$Recall={\displaystyle \frac{M}{N_r}}$$

(5)

$$F1 Score=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(6)

where

M: number of matching n-grams between the generated text and reference text;

N_g: Total number of n-grams in the generated text;

N_r: Total number of n-grams in the reference text;

Precision: the ratio of matched n-grams to the total n-grams in the generated text;

Recall: the ratio of matched n-grams to the total n-grams in the reference text;

F1 Score: the harmonic mean of precision and recall, which is used to balance both metrics;

4.3.3. Evaluation Results

To provide a consistent standard for BLEU scoring, reference answers were created using verified content from Wikipedia, addressing core topics on the Qin–Hang metallogenic belt, such as geological background, structural features, resource distribution, mineralization mechanisms, and controlling factors, as shown in

Table 2. Professional questions and answers to the standard library.

Questions	Answers
Metallogenic Characteristics of Porphyry Copper Deposits in the Qin–Hang Metallogenic Belt	Porphyry copper deposits in the Qin–Hang metallogenic belt are distributed across the northern, central, and southern sections, with primary mineralization occurring during the Yanshanian period. The porphyries associated with mineralization in the Qin–Hang belt predominantly belong to the calc-alkaline magma series, with rock types including granodiorite porphyry, granite porphyry, and subrhyolitic porphyry. These mineralized porphyries exhibit trace element characteristics similar to those of Neoproterozoic island arc igneous rocks, suggesting they originated from the partial melting of a mixture of asthenosphere-derived magma and middle-lower crustal material. This is indicative of crust-mantle contamination, likely linked to lithospheric mantle influenced by earlier subduction. The geodynamic setting of the Yanshanian porphyry copper deposits is closely associated with the subduction of the Pacific plate, while the arc magmatic features of the porphyry and copper deposits are related to pre-Mesozoic tectonic events in South China. The calc-alkaline porphyries in the Qin–Hang belt derive from partial melting of basaltic rocks at the base of a Proterozoic island arc (lower crust) during the Mesozoic, fundamentally carrying island arc subduction-related characteristics.
Regional Geological Background of the Qin–Hang Metallogenic Belt	The main extent of the Qin–Hang metallogenic belt corresponds to the Qin–Hang junction zone, a multiblock collisional zone formed by the collision and amalgamation of the Yangtze and Cathaysia blocks during the Jinningian period. The belt stretches approximately 2000 km from Qinzhou Bay in Guangxi to Hangzhou Bay in Zhejiang, with an overall S-shaped arc pattern. In the south, it is bounded by the Pingxiang-Guangfeng-Shaoxing fault separating it from the Cathaysia block, while in the north, the Yifeng-Jingdezhen-Shexian fault marks the boundary with the Yangtze block. The eastern segment of the Qin–Hang belt includes the entirety of the eastern Qin–Hang junction zone, along with Yanshanian-period magmatic mineralization zones on either side, including the eastern segment of the Jiangnan orogenic belt to the north and the Wugongshan-Tiantai uplift zone to the south. This region spans approximately 900 km in length and 190–230 km in width, covering an area of approximately 180,000 square kilometers. The basement strata of the eastern segment are Neoproterozoic in age, including formations such as the Shuangqiaoshan, Zhangcun, Xikou, Wannian, Shuangxiwu, Chencai, Yifeng, Tianli, and Tieshajie groups, which collectively represent a suite of marine volcanic-sedimentary deposits. Since the Neoproterozoic, the Qin–Hang belt has undergone multiple tectonic events. The Jinningian period involved oceanic-arc subduction and continental collision, leading to the convergence of the Yangtze and Cathaysia blocks along the Northeast Jiangxi major fault, forming the Jiangnan orogenic belt in the northern Qin–Hang belt, with a complete trench-arc-basin system. The Caledonian orogeny resulted in the complete amalgamation of the Yangtze and Cathaysia blocks, creating the unified South China block and broadly shaping the structure of the Qin–Hang junction zone and its surroundings. During the Hercynian period, intracontinental rifting along the western side of the Northeast Jiangxi deep fault, between the Yifeng-Jingdezhen-Shexian fault and the Pingxiang-Guangfeng-Shaoxing fault, formed the Zhejiang-Jiangxi depression zone (Pingxiang-Qiantang depression). This was followed by intense intracontinental folding and orogeny during the Indosinian period, leading to the formation of the Wugongshan-Tiantai uplift zone, resulting in the east segment of the Qin–Hang metallogenic belt’s characteristic “two uplifts with one depression.” The Yanshanian period witnessed a tectonic shift involving lithospheric extension and thinning, triggering large-scale granite magmatism and a surge in magmatic mineralization.
Distribution of Deposit Types in the Qin–Hang Metallogenic Belt	The distribution of deposits in the Qin–Hang metallogenic belt follows a distinct pattern, significantly influenced by the internal structure and evolutionary heterogeneity of the Qin–Hang junction zone. Yongzhang Zhou and colleagues proposed a division of the belt into northern, central, and southern sections based on the regional distribution of dominant ore types. The northern section is rich in copper, iron, and precious metals, with representative deposits including the Dexing copper deposit, Lingshuikeng lead-zinc-silver deposit, Yinzhan gold-copper-lead-zinc deposit, Jinshan gold deposit, and Ping Shui copper deposit, among others. The central section corresponds to the Nanling metallogenic belt, known for its abundance of granite and as a major source of tungsten-tin polymetallic minerals. Key deposits include Dongpo, Shuikoushan, Xitian, Xihuashan, Dajishan, Tongshanling, Huangshaping, and Shizhuyuan, all associated with magmatic activity. The southern section is primarily known for gold, silver, and polymetallic deposits, with major resources located in the Yunkaidashan, Dayao, and Shiwan mountains. Representative deposits include the Hetai gold deposit, Pangxidong silver deposit, Daming Mountain gold-silver deposit, Yun’an Gaochang lead-zinc-silver deposit, and the Changkeng large silver-gold deposit. Jingwen Mao et al. categorized the Qin–Hang belt and adjacent deposits into two metallogenic series and three subseries: Neoproterozoic Seafloor Exhalative-Sedimentary Copper-Zinc Deposit Series, and Yanshanian Granite-Related Tungsten-Tin-Copper-Lead-Zinc Polymetallic Deposit Series. The latter is further divided into: The Middle-Late Jurassic Porphyry-Skarn-Hydrothermal Vein Copper Polymetallic Subseries, The Late Jurassic Granite-Related Tungsten-Tin Polymetallic Subseries, and The Cretaceous Epithermal Low-Temperature Gold-Silver-Lead-Zinc-Tungsten-Tin Subseries associated with subvolcanic activity. Mao proposed that the Neoproterozoic massive sulfide copper-zinc deposits formed in a back-arc or fore-arc basin setting, while large-scale tungsten-tin polymetallic mineralization approximately 160–150 Ma is related to major granite magmatic events. After 135 Ma, mineral resources were concentrated in volcanic and fault-basin settings. Deming Xu and colleagues identified seven metallogenic series, including the Meso-Neoproterozoic seafloor exhalative-sedimentary copper-polymetallic deposits, Neoproterozoic marine sedimentary-metamorphic iron-manganese deposits, Paleozoic marine sedimentary-superimposed copper-lead-zinc-iron-manganese deposits, Caledonian granite-related tungsten-molybdenum-gold-silver polymetallic deposits, Indosinian granite-related tungsten-tin-niobium-tantalum-uranium polymetallic deposits, Yanshanian granite-related copper-lead-zinc-gold-tungsten-tin polymetallic deposits, and gold-silver deposits associated with regional dynamic metamorphic hydrothermal activity.

. These questions were reviewed and curated by experts to ensure comprehensive coverage of the key geological knowledge and mineralization rules.

Each model’s generated content was then evaluated sentence by sentence against the reference answers, with precision, recall, and F1 score calculated. Scores range from 0 to 1, with higher scores indicating stronger similarity between the generated text and the reference answer, thus objectively reflecting each model’s ability to generate accurate and relevant responses in this specialized field.

Evaluation Results Analysis Based on the Professional Background of the Qin–Hang Metallogenic Belt (

Table 3. Assessment results.

Model	Precisionaverage	Recallaverage	F1 Scoreaverage
ChatGLM3-6B	0.7916	0.8022	0.7969
ChatGLM3-6B RAG	0.8663	0.8421	0.8535
ChatCPT-4o	0.8507	0.8521	0.8514
Bing	0.8456	0.8145	0.8298
Gemini	0.8152	0.8140	0.8146

).

Precision Comparison: The ChatGLM3-6B RAG model achieves the highest precision score at 0.8663, indicating that integrating a specialized knowledge base for the Qin–Hang metallogenic belt significantly improves the accuracy of responses, especially in detailing structural features, mineralization mechanisms, and deposit characteristics. This demonstrates the substantial impact of a specialized knowledge base on response accuracy. ChatGPT- and Bing followed with precision scores of 0.8507 and 0.8456, respectively, indicating strong accuracy as general models. However, they lack the depth needed for highly specialized geological topics. Gemini scored 0.8152, which was slightly better than that of the base ChatGLM3-6B (0.7916) but still lower than that of the models with enhanced knowledge. The base model’s lower precision highlights that general knowledge alone is insufficient for high-quality content in specialized geological contexts.

Recall Comparison: The ChatGLM3-6B RAG model’s recall score of 0.8421 indicates comprehensive coverage of essential Qin–Hang metallogenic belt knowledge, including geological background, deposit distribution, and mineralization processes. Such comprehensive information is crucial for accurately representing the belt’s geological evolution. ChatGPT-4o’s recall of 0.8521 is close but lacks specific support for detailed topics such as precise stages of mineralization due to the absence of domain-specific data. Bing and Gemini scored similarly at 0.8145 and 0.8140, reflecting limited breadth in covering detailed geological structures and deposit types. The base ChatGLM3-6B model scored only 0.8022 in recall, underscoring the importance of a knowledge base in capturing detailed domain-specific information.

F1 score comparison: With an F1 score of 0.8535, the ChatGLM3-6B RAG effectively balances precision and recall, utilizing the knowledge base to provide accurate, complete responses. This version offers reliable information for complex topics such as mineralization mechanisms, geological evolution, and resource distribution. ChatGPT-4o’s F1 score of 0.8514 indicates balanced responses, although it lacks the depth of knowledge-enhanced models. Bing and Gemini scored 0.8298 and 0.8146, respectively, achieving a compromise between accuracy and coverage, but their lack of specialized knowledge limits their performance on complex geological questions. The base ChatGLM3-6B model has the lowest F1 score at 0.7969, highlighting the challenges of achieving balanced, high-quality responses without a specialized knowledge base.

Overall, ChatGLM3-6B outperforms the other models in terms of precision, recall, and F1 score, particularly in addressing detailed questions concerning the geological background, deposit types, and mineralization mechanisms of the Qin–Hang belt. This underscores the effectiveness of a specialized knowledge base in enhancing response quality in professional domains. While ChatGPT- is stable as a general model, it lacks the detail needed for specialized geological topics. Bing and Gemini offer similar performance but lack accuracy and coverage in complex topics, such as mineralization processes and structural evolution. The base ChatGLM3-6B model scored lowest in all the metrics, especially compared with its knowledge-enhanced version, emphasizing the essential role of a knowledge base in achieving high accuracy and coverage in specialized fields.

5. Discussion

This study serves as an initial exploration into developing an intelligent conversational knowledge base for the Qin–Hang metallogenic belt via the ChatGLM3-6B large language model, with a focus on assessing the model’s performance when enhanced with a specialized knowledge base. While this approach demonstrated notable improvements in generation quality, as an early-stage exploration, certain limitations and improvement areas were identified. First, the current Qin–Hang metallogenic belt knowledge base remains limited in scope. There is significant potential for expanding and refining this belt to more comprehensively cover the geological characteristics, mineralization processes, and regional evolution specific to this belt. Moreover, as knowledge base construction mainly relies on publicly available data and literature, future involvement from domain experts could further enhance its accuracy and relevance. In terms of evaluation, this study utilized BLEU scores alongside precision, recall, and F1 scores to assess generation quality. While these automated metrics provide valuable insights into content quality, they may not fully capture the depth of professionalism and practical applicability required in specialized domains. The incorporation of additional evaluation methods, such as human expert assessments, could provide a more comprehensive evaluation of the model’s domain-specific performance.

In conclusion, building an intelligent knowledge base for the Qin–Hang metallogenic belt presents an effective pathway for improving the domain-specific performance of large language models tructuring and integrating specialized knowledge into the model significantly enhances generation quality and domain accuracy, which is especially crucial for fields such as geology that require advanced expertise [97]. Future research should focus on expanding the knowledge base’s breadth and depth and refining its integration with the generative model to increase its applicability in complex domains. Additionally, developing an automatically updated knowledge base could help keep pace with ongoing geological research, ensuring that the generated content remains current and relevant. Further exploration into incorporating multimodal data, such as geological maps and remote sensing images, may also enrich the model’s interpretive and generative capabilities in handling complex geological information.

In light of the recent emergence of various geoscience-focused LLMs, such as GeoGPT [38], GeoGalactica [41], and K2 [40], this study provides a comparative analysis to better contextualize its contributions. As shown in

Table 4. A comparison of representative geoscience large language models in terms of architecture, language support, geographic focus, openness, QA capabilities, and evaluation.

Comparison Dimension	GeoGPT	GeoGalactica	K2	ChatGLM3-6B RAG (This Study)
System Architecture	KG + GPT-based QA	Scientific generative LLM	Foundation model for geoscience	RAG-based QA (LangChain + ChatGLM)
Language Support	English only	English only	English only	Bilingual (Chinese + English)
Geographic Focus	Global terminology	Global scientific corpus	Multi-region remote sensing	Focused on the Qin–Hang metallogenic belt
Openness	Partially open (no weights)	Structure open, no weights released	Fully open (code + data)	Fully open (code + corpus structure)
QA Capability	Entity recognition & basic queries	Generative QA (no benchmark)	No QA module	Vector retrieval + generative answers
Performance Evaluation	Not reported	Not reported	Evaluated on GeoBench	Evaluated using F1/Precision/Recall

, unlike existing systems that primarily rely on English-language corpora or structured knowledge graphs, our system introduces a bilingual (Chinese–English) domain-specific QA corpus centered on the Qin–Hang metallogenic belt. Leveraging the LangChain framework and the ChatGLM3-6B model, we implement a modular and reproducible retrieval-augmented generation (RAG) pipeline tailored for geological question answering.

In addition to its flexible architecture, the proposed system demonstrates notable performance gains over general-purpose models (e.g., GPT-4o, Bing, Gemini) in terms of domain-specific precision. Furthermore, its integration of source-grounded responses enhances interpretability and transparency, making it more aligned with the expectations of scientific reasoning. All source code, configuration files, and evaluation datasets have been fully open-sourced, facilitating reproducibility and broader adoption in academic and industrial settings.

Despite these contributions, several limitations remain. First, the current knowledge base is statically constructed, lacking mechanisms for automated updates from ongoing geological research. Second, while the corpus is built from public scientific literature, it currently lacks expert-validated ontologies or structured domain knowledge, which limits its ability to handle deep reasoning tasks involving geological semantics and terminology. Additionally, the system does not yet support multimodal inputs such as geological maps, stratigraphic sections, or remote sensing imagery—elements that are increasingly important in practical geological interpretation tasks.

Future work may focus on incorporating multimodal fusion between textual and visual modalities, integrating expert knowledge graphs into the pipeline, and testing the system’s adaptability to other metallogenic zones. These directions will further enhance the domain robustness, interpretability, and practical utility of AI-driven geological QA systems.

6. Conclusions

This study leverages the ChatGLM3-6B large language model to construct a preliminary intelligent conversational knowledge base for the Qin–Hang metallogenic belt and evaluates its performance. By integrating retrieval-augmented generation (RAG) technology, this study investigates how a specialized knowledge base can enhance the effectiveness of large language models in specific domains. The results indicate that the ChatGLM3-6B model, augmented with the Qin–Hang metallogenic knowledge base, achieves a significantly higher generation quality, with an F1 score of 0.8689, outperforming the base version without the knowledge base (0.7969) and other mainstream large language models, including ChatGPT-4 (0.8514), Bing (0.8298), and Gemini (0.8146). The combined ChatGLM3-6B model demonstrates superior generation accuracy and content coverage for the Qin–Hang metallogenic belt, particularly in addressing complex questions on mineralization mechanisms, geological background, and ore distribution. These findings suggest that integrating a specialized knowledge base with RAG technology effectively addresses the limitations of general language models in domain-specific knowledge, enhancing the authority and precision of generated responses.

Future work will focus on expanding the scope and depth of the knowledge base, refining the integration between the knowledge base and the language model, and incorporating evaluations from human experts for a more comprehensive assessment. These advancements aim to strengthen the model’s applicability in complex professional fields, laying a solid foundation for intelligent conversational systems in geological research and applications.