Building an Intelligent QA System for Smart City Planning: Integrating LLMs and Knowledge Graphs

Abstract

Smart city planning involves a wide range of knowledge domains. However, general intelligent Question Answering systems often fall short when applied to this domain, and the relevant studies are not yet sufficient. To this end, this paper constructs an intelligent QA system that combines a large language model with a domain-specific knowledge graph. Capable of understanding questions accurately and generating professional answers, this system is designed to provide efficient knowledge services for smart city planning by following four steps. First, based on four authoritative planning guidelines, a domain-specific knowledge graph with a four-layer framework is constructed using Neo4j Community Edition 5.26.24. The framework includes top-level goals, knowledge modules, standard terminology and community scenarios. Subsequently, natural language questions are classified and matched with the templates before being converted into structured queries. Finally, the system performs Cypher query language queries and invokes ChatGLM4 to generate professional answers. The knowledge graph contains 100 entity nodes and 44 relations, and its ontology layer defines 28 entity types and 12 relation types. Therefore, the domain knowledge is structured and visualized, and planning professionals can intuitively retrieve diverse planning elements. In addition to its intelligent knowledge query function, this system assists planning professionals in preparing planning schemes and verifying compliance, reducing the time spent on reviewing regulations and comparing clauses, improving the efficiency of scheme preparation, and facilitating the refined implementation of urban renewal projects. It has high application value in smart city planning practices. Its construction approach can also serve as a reference for intelligent knowledge services in other fields.

1. Introduction

1.1. Background and Significance

Smart city planning is Information and Communication Technology (ICT)-driven. Revolving around systematic, structured and intelligent models and tools, it responds to residents’ demands and allocates urban resources with the purpose of achieving equitable and sustainable development while staying people centered and problem oriented [1]. Urban planning professionals and policymakers often need to acquire precise knowledge, including planning terminology, technical standards, indicator criteria and case paradigms, in their routine work. However, there are no dedicated intelligent Question Answering (QA) tools at their disposal. Although general-purpose large language models (LLMs) are capable of understanding and generating natural language, they can produce plausible yet incorrect content due to the absence of authoritative domain-specific constraints in training data. This undermines the credibility of the answers [2,3]. While knowledge graph-based QA systems enable structured and traceable knowledge retrieval, they require users to learn specific query languages, such as SPARQL Protocol and RDF Query Language (SPARQL), which imposes a high barrier on non-professional users [4]. Moreover, most existing QA systems are designed for general domains, and the exploration of domain-specific QA systems is in its infancy, with few mature in-depth applications. Therefore, constructing a smart city planning QA system that integrates LLMs with knowledge graphs, which balances domain expertise with user-friendly interaction, possesses both theoretical significance and practical value.

1.2. Current Research Status

Over the past few years, the combined use of large language models and knowledge graphs has become a research hotspot in artificial intelligence [5]. In vertical-domain QA applications, this combined architecture has delivered promising outcomes. For instance, Tan Ping et al. developed a depression QA system that integrated Chinese and Western medicine. They used knowledge graphs to constrain the LLM’s output and thus ensured professional credibility [6]. Nevertheless, the research on intelligent QA systems for smart city planning is still at an early stage. Most existing efforts focus on querying urban data and enabling semantic interoperability. Deng et al.’s smart city QA system built on The World Avatar ontology [4] is such a case. Studies on professional QA tailored to the norms and standards of smart city planning remain limited. Therefore, it is necessary to construct a dedicated, integrated QA system for smart city planning by drawing on experience from healthcare and other fields.

1.3. Research Content and Technical Approach

1.3.1. Core Research Questions

Currently, both general-purpose large-scale models and independent knowledge graphs have their shortcomings in intelligent planning question answering scenarios: the former are prone to generating factual illusions, while pure knowledge graphs have rigid output formats, and existing tools are insufficient to support planners in carrying out scheme preparation work. Based on these industry pain points, this paper focuses on three core research questions:

(1)

How can a hierarchical domain knowledge graph be constructed based on existing national planning standards to achieve structured storage, source tracing, and visualization of planning provisions?

(2)

After integrating knowledge graph retrieval strategies with a large language model optimized based on prompting engineering, can the generation of illusions by large-scale models be suppressed, and can the accuracy and compliance of question answering results be improved?

(3)

Can the constructed Knowledge Graph–Large Language Model (KG-LLM) integrated QA system assist in planning scheme preparation and compliance review, and improve the efficiency of planning practice?

1.3.2. Research Objectives and Technical Approach

To address the research questions, this paper focuses on the construction of a KG-LLM fusion question answering framework, establishes the hierarchical research objectives, and forms a closed-loop technical implementation path for data preprocessing, knowledge graph construction, system development, and comparative evaluation.

First, using authoritative planning guidelines as the data source, the core domain knowledge was extracted through text preprocessing. A four-layer hierarchical knowledge graph was constructed using manual annotation and rule validation. This involved sequentially building the top-level goals, knowledge modules, standard terminology, and community scenarios. Neo4j was used to achieve structured storage, text traceability, and visual management of the knowledge graph, achieving the research objective of digital archiving of planning specifications.

Second, a fusion-based question answering architecture was built upon the existing knowledge base. A question classification and template matching mechanism was designed to achieve accurate knowledge graph retrieval, with a supporting vector knowledge base to handle template matching failure scenarios. Answer generation was achieved through hierarchical standardized prompts linked to the ChatGLM4 model. Authoritative structured knowledge constrained model generation bias, aiming to suppress factual illusions in large models and optimize answer compliance.

Finally, two baseline systems—a standalone knowledge graph-based system (KG-only) and a standalone large language model (ChatGLM4, LLM-only)—were set up and compared with our proposed hybrid scheme integrating the LLM with the knowledge graph. A test set of 20 smart planning-related questions was prepared to evaluate the system across multiple dimensions. Based on the evaluation results, this paper points out directions for optimizing both the knowledge graph and the QA system.

2. Related Theory and Technical Foundations

2.1. Large Language Models

ChatGLM4 (General Language Model 4, GLM4) was selected as the core generation engine for the system. Developed by Zhipu Artificial Intelligence (Zhipu AI), ChatGLM4 is a new-generation bilingual LLM that shows strong capabilities in deep semantic parsing, contextual reasoning and professional text generation [7]. Based on research combining publicly available benchmark data and industry evaluations from Zhipu AI [8], ChatGLM4 ranks among the top tier of domestic closed-source models in comprehensive performance across multiple authoritative benchmark tests, including integrated understanding, mathematical logic, common sense reasoning, and code analysis. It exhibits significant performance advantages in complex logical deduction and long-text semantic understanding tasks. Furthermore, the model features a 128K ultra-long context window, capable of fully accommodating multiple planning texts and structured triplet combinations from the knowledge graph. It demonstrates excellent recall accuracy in scenarios involving sequential reasoning and cross-regulatory clause comprehensive deduction, effectively reducing the factual illusion problem in regulatory Question Answering. From a task adaptation perspective, GLM4 has undergone multi-stage fine-tuning of professional corpora and, leveraging its built-in tools’ capabilities, can accurately connect with knowledge graph retrieval results. After customized domain-specific prompt word engineering, it can transform the structured data output from the graph into logically rigorous, industry-compliant natural language responses, perfectly matching the technical requirements of this research’s “Intelligent Question Answering for Planning Integrating Knowledge Graphs and Large Models” technical route.

2.2. Knowledge Graph Technology

A knowledge graph refers to a knowledge representation method that uses a graph structure to capture the semantic relations among entities. It is fundamentally composed of triples in the forms of “entity–relation–entity” or “entity–attribute–attribute value” [9]. In this research, Neo4j was used as the underlying storage and retrieval engine, where the nodes represented entities and the edges represented semantic relations. The Cypher Query Language (Cypher) declarative query language was employed to support exact matching, path queries and multi-hop relational retrieval [10].

Building a knowledge graph typically involves four core stages: knowledge extraction, knowledge fusion, knowledge processing and knowledge updating. During extraction, the entities, semantic relations and attribute parameters are identified from national standards, either automatically or semi-automatically. Knowledge fusion aligns the coreferences, resolves conflicts, and removes redundancies across multiple standard sources. Knowledge processing goes a step further: it goes through ontology construction, reasoning and quality assessment to building a structured knowledge system. Knowledge updating ensures the timeliness of the generated knowledge by performing incremental or full updates at both the schema and data levels. Together, these stages ensure the consistency, authority and completeness of the knowledge base, providing reliable and controllable factual support for the QA system built on top of it [11].

2.3. Knowledge Characteristics and Data Foundation for Smart City Planning

The knowledge required in smart city planning and urban renewal features complex hierarchies, cross-domain intersections and strong statutory constraints. Any output must comply with official technical standards at a national level.

Four national-level standards and guidelines related to smart city and urban renewal were selected as the sole authoritative knowledge sources. They cover the full spectrum of technical requirements, from top-level architecture design and general knowledge models to specialized terminology and community living circle implementation. By performing structured ontology modeling and fully extracting the entities and relations from these original regulatory texts, this study developed a dedicated domain-specific knowledge graph. This approach filters out low-quality information, such as online noise and unofficial interpretations, at the source, fundamentally suppressing the tendency of LLMs to hallucinate, thus ensuring that all system output is verifiable, compliant and traceable.

3. Core Design of the Smart City Planning QA System

3.1. System Overview

This study developed a professional QA system for smart city planning by integrating a large language model with knowledge graphs. Based on four authoritative planning guidelines, this study constructed a domain-specific knowledge graph with a four-layer framework, covering top-level goals, knowledge modules, standard terminology and community scenarios. A fallback vector knowledge base was introduced to extend the knowledge coverage of the system. The system consists of three modules: question processing, knowledge graph retrieval and answer generation [12]. As shown in Figure 1, once a user inputs the natural language questions, the system classifies the questions and matches them with the template. The system then generates structured Cypher queries to retrieve data from the planning knowledge graph. If precise matching is successful, authoritative data, such as entities and relationships, will be extracted from the knowledge graph. If the matching fails, automatic degradation will be triggered to initiate fallback retrieval based on multiple knowledge sources and vector knowledge bases. The knowledge retrieved from both paths is then packaged into standardized constraint prompts and fed into ChatGLM4 LLM, which generates compliant, traceable, and professionally fluent natural language answers.

3.2. Data and Knowledge Preparation

3.2.1. Core Data Sources

The system’s knowledge base was built on the following four authoritative regulatory documents: GB/T 36332-2018, Smart city—Domain knowledge model—Core conceptual model [13]; GB/T 36333-2018, Smart city—Top-level design guide [14]; GB/T 37043-2018, Smart city—Terminology [15]; and TD/T 1062-2021, Technical guideline for community living circle planning [16]. The first three are national standards and the last one is a planning technical guideline. These documents jointly provide an authoritative and theoretical basis for the system.

3.2.2. Data Preprocessing

The four regulatory documents were subjected to structured preprocessing to extract the core knowledge fragments, following a hierarchical logic: guideline name → core module → key information → source section. The terminology was defined according to GB/T 37043-2018. Duplicate content across guidelines prioritized core specifications, entity names and relational expressions were standardized, and content completeness was validated for the core sections of each guideline. Over 300 valid knowledge fragments were extracted, after eliminating redundant background descriptions and irrelevant case narratives.

3.2.3. Knowledge Requirement Analysis

To satisfy typical QA scenarios in smart city planning, this study divided the knowledge requirements into four core dimensions, according to the structured characteristics of the knowledge system. Such a classification reflects the progressive logic of planning knowledge from macro strategy to micro implementation. It also provides a direct basis for entity definition and relationship design for subsequent knowledge graphs and functional implementation of the QA system [17].

(1)

Top-level design

This dimension covers the overall development goals, architecture, top-level implementation principles, key renewal projects, diverse operation modes, and full-cycle guarantee measures of smart city planning. The relevant knowledge, on the whole, forms the macro framework of the planning system and serves as the foundation for understanding the planning logic and policy direction.

(2)

Terminology query

This dimension focuses on professional vocabularies and definitions, covering basic general terms, framework and model terms, data resource terms, infrastructure terms and supporting technology terms. Such knowledge enables users to accurately understand planning documents, technical standards and design specifications.

(3)

Knowledge association

This dimension contains hierarchical subordinate relations (e.g., “community living circle—basic security service—basic convenience facility”), support–adaptation relations (e.g., “Internet of Things—urban smart sensing facilities”), and rigid constraint relations (e.g., “FAR upper limit—building height control requirement”). The corresponding knowledge supports deep multi-hop reasoning across the concepts and standards, which is key to realizing complex semantic understanding.

(4)

Community planning

Centering on urban renewal and old neighborhood renovation scenarios, this dimension formulates the planning criteria for community living circles, such as hierarchical allocation, public service supply, element type definition, spatial layout specification, disaster resilience requirements and environmental quality improvement measures. It is oriented toward grassroots planning implementation and can directly support practical queries from planning professionals and managers.

These four knowledge dimensions define the coverage and semantic depth of the knowledge graph, and set clear capability boundaries for the subsequent ontology design, entity–relation extraction, and the retrieval and generation strategies of the QA system.

3.3. System Construction Methodology

This section describes the design and implementation of each system module following the main workflow: user query input → question classification and routing → knowledge retrieval → answer generation.

3.3.1. Knowledge Graph Construction

Ontology Layer Construction

The ontology layer in smart city planning is a standardized semantic abstraction of the consensual professional knowledge in urban planning and renewal. It defines the unified entity categories, semantic relations and logical constraint rules, thereby constructing a standardized hierarchical knowledge system to support intelligent QA and knowledge reasoning [18]. By systematically analyzing the structures, content attributes, and knowledge classification of the four regulatory documents, this study designed an ontology framework for smart city planning, as shown in Figure 2. This framework consists of four layers, namely the core foundation layer, the top-level design application layer, the community living circle application layer and the cross-domain general layer, totaling 27 entity types and 12 relation types.

(1)

Core foundation class (derived from GB/T 36332-2018 and GB/T 37043-2018): This layer includes the knowledge model base class and its seven subclasses, namely the entity, service, situation, measurement, event, role and collaboration base classes, as well as the standard terminology and its subclasses (data resource terms, infrastructure terms, and supporting technology terms). It lays a unified theoretical foundation and terminology support for the whole knowledge graph.

(2)

Top-level design application class (derived from GB/T 36333-2018): This layer involves six entity types, namely the top-level design goals, overall architecture dimensions, top-level design principles, key projects, operation models and guarantee measures. These correspond to the macro control and system design of urban planning and renewal.

(3)

Community living circle application class (derived from TD/T 1062-2021): This layer covers six entity types, namely the community living circle levels (town level, 15 min/5–10 min tiers; and village/group tiers), service elements (including six subcategories, such as community services and employment guidance), element types (basic security, quality improvement, and characteristic guidance), disaster prevention elements, layout requirements and environmental improvement elements. It targets practical implementation scenarios at the grassroot community level.

(4)

Cross-domain general class: This layer contains four entity types, namely the time information, spatial information, indicator types (scale, coverage, efficiency and quality) and knowledge sources. These serve as a universal link connecting all the ontology modules.

This research also designed customized standardized attribute fields for the different entity categories. For the service element entities, attributes such as service content, scale indicator, coverage indicator and quality indicator were configured. The top-level design goal entities were assigned definitional attributes like goal type and goal connotation, while the knowledge source entities were equipped with traceability attributes, including standard number and corresponding section. This guarantees accurate traceability and source verifiability across the entire knowledge chain.

At the semantic relation level, 12 standard relation types were defined and grouped into four categories: (1) hierarchical subordinate relation (“is a kind of”, “contains subclass”, “contains goal”); (2) underlying support relation (“supports top-level design”, “governs community elements”, “quantifies target indicators”, “describes entity”); (3) scenario application relation (“belongs to architecture dimension”, “configures level elements”, “meets layout requirements”, “depends on guarantee measures”); and (4) cross-scenario coordination relation (“bidirectional adaptation”).

Knowledge Extraction and Fusion

The regulatory documents on smart city planning contain highly specialized concepts with complex hierarchical structures and intricate logics. This research adopted a manual extraction approach, supplemented by rule-based validation, in order to identify, screen and extract data from the preprocessed regulatory corpus [19]. Guided by the ontology hierarchy, this study extracted entities, semantic relations and attributes from the texts. Then, they were aligned with multi-source entities and relations, redundancies were eliminated, and they were sorted into standardized triples stored in a dedicated knowledge base. Domain experts performed cross-validation sampling to ensure the normative accuracy and authority of the entire knowledge system, providing reliable and trustworthy data support for the subsequent intelligent QA and reasoning.

Knowledge Storage and Visualization

Neo4j was used as the graph database for knowledge storage. Compared with traditional relational databases, Neo4j can better represent the multi-dimensional nature of knowledge networks related to smart city planning, from concepts and attributes to entity relations [20]. The triple data were imported into Neo4j through writing Cypher statements to construct the knowledge graph step by step. To be specific, entities were mapped as independent nodes, semantic relations as directed edges between nodes, and various attributes were stored as key–value pairs. After that, the Cypher query language was applied to realize entity retrieval, relation matching and multi-hop relational queries. At the same time, Neo4j’s native visualization interface enabled an intuitive display of knowledge network, which made it easier to verify the completeness and logical soundness of the knowledge system. The visualized results of each layer in our four-layer ontology architecture are shown in the figures below (see Figure 3, Figure 4, Figure 5 and Figure 6).

Knowledge Graph Topology Quantization and Cluster Feature Analysis

Based on the four-layer ontology architecture and the basic data of graph entities and relationships, this study conducted a topological quantitative analysis from two dimensions, classic centrality indicators and community clustering characteristics, clarifying the spatial distribution patterns of knowledge in the field of smart city planning:

(1)

Centrality Indicator Analysis: Three classic graph indicators—degree centrality, betweenness centrality, and proximity centrality—were selected for measurement. The degree centrality results showed that the top-level specifications and standard terminology nodes were the core hubs of the network, while the national standards, such as the “Smart City Master Planning Guidelines”, and community-supporting element nodes played a crucial connecting role in the entire graph. The betweenness centrality indicated that cross-domain general entities, such as space and indicators, were important bridges connecting the top-level planning and implementation scenarios. The standard terminology nodes had the best proximity centrality, serving as a multi-branch common entry point, explaining the efficient internal mechanism of terminology-related question retrieval from a structural perspective.

This graph contains 100 entities and 44 sets of triplet relationships, with an average node degree of 0.88, consistent with the sparse network characteristics of small-scale graphs in subdivided fields.

(2)

Community Cluster Analysis: Unsupervised community mining was performed using the Louvain algorithm. The experimental results divided the graph into four major communities: top-level design, standard terminology, community life circles, and cross-domain general knowledge. These communities matched the pre-built four-layer ontology architecture, indirectly confirming that the ontology framework aligned with the natural clustering distribution patterns of planning knowledge.

(3)

Clustering Coefficient Analysis: The overall average clustering coefficient of the graph was low, but the clustering coefficients within each cluster were relatively higher. This reflects the structural characteristics of closely related knowledge within the same category and the cross-domain connectivity among different knowledge clusters through a small number of intermediary entities. This aligns with the industry attributes of planning knowledge, which is domain-specific and relies on standardized indicators for interoperability.

3.3.2. Question Processing Module

The question processing module is designed to meet the practical needs of planning professionals. It accurately parses users’ natural language queries, intelligently identifies the question types, and matches the corresponding retrieval and reasoning logic. These functions enable planning professionals to acquire professional knowledge efficiently and precisely and help with scientific decision-making [21].

Question Classification

Based on cognitive complexity, knowledge depth and typical application scenarios, the user queries are classified into three core types: factual definition, relational constraint, and inferential advisory (see

Table 1. Question classification.

Question Type	Example Question
Factual definition	Which data resource terms are essential to establishing the “current status database” in urban renewal?
Relational constraint	Based on the relationship between the “service elements” and “layout requirements” in the knowledge graph, and the allocation relationship between “community living circle levels” and “service elements”, the following questions are derived: If an idle factory is converted into a “community health service center”, what “layout requirement” attributes must be satisfied? Which living circle level should this facility belong to?
Inferential advisory	With the growing popularity of remote work and online education, virtual services may have less need for physical commercial outlets, but increased need for new facilities, such as community-shared live-streaming rooms and express stations. Based on the allocation logic of “quality improvement” elements and the driving effect of “supporting technology terms”, the adjustment of proportion between “non-digitalizable” core facilities and “partially replaceable” facilities is explored in future urban renewal.

).

This study used a lightweight rule-based classifier that relied on trigger word lists and sentence pattern features to automatically classify user questions. For example, factual definition questions often contain trigger words like “define”, “what is” and “indicator”. Relational constraint questions feature expressions such as “relation”, “composed of” and “includes”. Inferential advisory questions often start with words like “how to”, “whether”, “evaluate” and “trend”. In cases where some complex questions with vague features cannot be precisely matched according to the abovementioned rules, the system defaults them to the inferential advisory category to ensure that all query needs are effectively covered.

Question Processing Routing

To deal with high-frequency planning scenarios, this study predefined three types of standard question paradigms, each with an exclusive query logic template for the knowledge graph [22]. As shown in Figure 7, after classifying the user’s questions, the system distributes them to corresponding processing paths according to the matching results.

Factual definition: The system matches a predefined terminology query template. For static questions about concepts and indicators, it implements precise node attribute retrieval and directly returns the relevant original regulatory text.

Relational constraint: The system matches an entity–relation query template and performs multi-node and multi-level graph path retrieval to clarify the hierarchical structure and constraint logic among planning elements.

Inferential advisory: Instead of using a fixed template for exact retrieval, the system initiates a multi-source knowledge recall process. It extracts from the knowledge graph all neighboring information related to the question, including the core concepts, principle clauses and hierarchical elements. This provides a complete knowledge base for the LLM to conduct compliant reasoning. If inquiries involving emerging concepts have not yet been covered in the existing knowledge graph, the system will perform analogical reasoning based only on existing planning principles.

In addition, for factual definition or relational constraint queries where template matching fails or the intent is ambiguous, the system also redirects them to the inferential advisory processing flow. There, the integrated KG-LLM framework is used to generate in-depth and comprehensive answers, which supports planning professionals with their knowledge queries and scientific decision-making.

3.3.3. Knowledge Graph Retrieval Module

Neo4j’s Cypher query language is adopted for knowledge graph retrieval. Cypher offers concise and user-friendly syntax, and efficiently supports precise querying of entities, attributes and relations in the graph database. This enables the system to possess stable and reliable knowledge access capabilities [23].

Neo4j-Based Precise Retrieval

For factual definition and relational constraint queries, the system connects to Neo4j through a program interface, based on the question classification results and the preset query logic. It executes Cypher statements to accurately extract the deterministic knowledge, including entity attributes, hierarchical relations and configuration requirements. After retrieval, the system parses the returned node, relationship and attribute information in a structured manner, and converts the standardized graph data into natural language segments to generate standardized, accurate and authoritative answers.

Multi-Source Knowledge Retrieval

For complex inferential advisory planning questions, the system uses a multi-source knowledge recall strategy to provide a reliable basis for LLM generation. The process is as follows:

(1)

Core concept extraction: The system identifies the key entities and core concepts from the user’s questions.

(2)

Graph association retrieval: The system searches the knowledge graph for corresponding concept nodes and neighboring associations, retrieving the relevant regulatory clauses, planning principles, hierarchical elements and constraint conditions.

(3)

Vector knowledge supplementation: A domain-specific vector fallback knowledge base is constructed based on the graph ontology and the four authoritative standards. The system conducts semantic similarity matching between the question text and the vector knowledge base at the same time to recall the Top-K most relevant principled clauses, so as to supplement the systematic background knowledge.

(4)

Knowledge fusion: The system integrates the structured graph knowledge with the vector text knowledge, and feeds them into the LLM as factual constraints for answer generation.

If the concept raised by the user is not covered by either the existing knowledge graph or the vector fallback knowledge base, the system will not generate an unsupported conclusion directly, nor will it refuse to answer the question. Instead, it will leverage ChatGLM4’s industry knowledge in the general domains regarding smart city and urban renewal, and conduct reasonable reasoning based on the underlying principles of analogous standards and common industry logic. This ensures that complex, emerging and cross-domain planning questions still receive complete and referable professional answers.

3.3.4. Answer Module Design

To achieve precise and controllable intelligent output, the answer module integrates a three-level flexible degradation prompt strategy with parameter control. In response to deterministic factual definition and relational constraint queries, the system employs a “strict traceability” fixed prompt. It requires ChatGLM4 to answer the questions strictly based on the 44 standard triples retrieved from the knowledge graph and the corpus, and prohibits the model from generating answers beyond the scope of the data sources. The system also requires that each piece of output precisely cite the corresponding national standard number, so as to ensure the authority of the advice. In response to complex inferential advisory planning issues, the system switches to a “normative reasoning” prompt. In the case that the knowledge graph lacks sufficient facts, the system will trigger an adaptive fallback mechanism. First, the system issues a “knowledge boundary declaration” to clarify non-standard sources, then it conducts compliant logical reasoning according to the authoritative knowledge documents, and distinguishes between “original regulatory text” and “model-extrapolated inference”. To reduce hallucinations, the system sets the model temperature parameter at 0.2, coupled with Top-P filtering [24]. This ensures that the output has certainty and rigor under a professional context.

With such a design, the system can accurately answer established facts based on the four core standards, while conducting well-supported inferential analysis on complex planning issues that go beyond existing regulatory provisions. In this way, it provides planning professionals with authoritative and insightful QA services.

4. System Experiments and Optimization

4.1. Experimental Design

4.1.1. Experimental Subjects, Baseline Settings and System Operating Environment

To evaluate the performance and advantages of the proposed LLM–KG integrated QA framework in a quantitative way, three schemes were set up for horizontal comparison:

Experimental group (proposed): The proposed framework is connected to the Neo4j planning knowledge graph and the vector backup knowledge base. It embeds a lightweight rule-based classifier for routing the three types of questions alongside the differentiated graph retrieval strategies, and uses layered prompt engineering to invoke the ChatGLM4 LLM. This enables the full-chain intelligent QA pipeline—from precise retrieval to compliant reasoning.

Baseline 1 (LLM only): A pure large language model baseline. It uses the original ChatGLM4 model without any external knowledge graph or regulatory knowledge bases. The users’ questions are directly fed into the model, which generates answers automatically.

Baseline 2 (KG only): A pure knowledge graph baseline. It relies only on the Neo4j knowledge graph to perform Cypher queries for precise retrieval, and returns structured query results in addition to original knowledge fragments. But this baseline does not involve any natural language refinement or inferential generation by the LLM.

In the early stages of the experiment, this study completed the topological quantization and clustering calculation of the knowledge graph. Based on the structural features of the graph, the Cypher retrieval path and the vector knowledge base classification logic were optimized, providing structural support for differentiated routing for the three types of questions.

Three sets of comparative experiments were conducted in a unified hardware and software environment. The hardware platform used an Alienware m16 R1 laptop with a 13th-generation Intel i9-13900HX processor, an RTX 4080 laptop mobile graphics card with 32 GB of RAM and 12 GB of VRAM, and running Windows 11.

The study used Visual Studio Code (Version 1.117.0) as the integrated development environment, relying on the Python 3.9 interpreter for system coding, program debugging, and experimental testing. Project dependency configuration and code execution were achieved through the software’s built-in terminal. For the software components, the knowledge graph was stored and retrieved using Neo4j Community Edition, and the ChatGLM4 model was remotely accessed via the official Application Programming Interface (API). The experimental groups shared the same hardware and software deployment conditions as the two baseline schemes to eliminate system errors caused by inconsistent experimental environments and to ensure the reproducibility of the experimental results.

4.1.2. Test Dataset

Based on the knowledge graph built on the four core planning standards, this study designed 20 test questions focusing on the pain points of smart city planning and urban renewal practices. Combining the distribution characteristics of real-world problems in the planning industry with the evaluation objectives of three sets of comparative experiments, a 6:6:8 question–type ratio was ultimately adopted: 6 questions on fact definition, 6 on relational constraints, and 8 on deduction and suggestion. The fact definition and relational constraint questions were used to balance the performance ceiling of pure knowledge graph solutions on deterministic retrieval tasks; the higher proportion of reasoning questions aligns with the reality of complex and comprehensive problems in urban renewal practices, facilitating a clear distinction between the performance gap between integrated solutions and single-baseline models in high-order deduction tasks. The three types of questions correspond to different scenarios in planning practice: the fact definition questions revolve around legal terms and clause interpretations, covering terminology definition scenarios, used to verify the accurate retrieval capability of the knowledge base; the relational constraint questions focus on hierarchical and clause constraint logic, corresponding to entity relationship sorting scenarios, emphasizing the assessment of knowledge graph relational query performance; and the reasoning and suggestion questions are geared towards comprehensive implementation scenarios, such as land optimization, street renewal, and disaster prevention and management, requiring the integration of multiple regulations for deduction, and are used to test the model’s comprehensive deduction and scenario adaptability capabilities. All test questions are expressed in everyday natural language, without any structured prompts, templates, or guiding clues, in order to simulate the real questioning habits of practitioners in practice and ensure the credibility and engineering reference value of the experimental conclusions.

4.1.3. Evaluation Metrics

Aligned with the core requirements of the smart city planning QA scenario, a five-dimensional evaluation framework was constructed. Each dimension is assessed on a 5-point Likert scale by human evaluators. The definitions are as follows [25]:

Accuracy captures the degree to which the answer aligns with the original regulatory text and its underlying principles, specifically targeting factual deviations and hallucination issues.

Completeness captures whether the answer exhaustively covers all the core points raised by the question, without omitting any key information.

Fluency captures the linguistic smoothness, the normative use of professional terminology, and the overall logical coherence.

Credibility captures whether the answer is verifiable and logically sound, and whether it maintains a clear distinction between the original regulatory statements and reasonable inferences.

Scenario relevance captures the degree to which the answer fits the practical context of smart city planning and urban renewal, reflecting its practical reference value.

4.1.4. Assessment Personnel and Implementation Process

The evaluation invited eight evaluators with dual-domain qualifications. Four were from the field of urban planning and urban renewal, possessing over three years of relevant policy research and project practice experience; the other four were from the field of natural language processing and artificial intelligence, with experience in evaluating question answering systems that integrate knowledge graphs and large language models, thus meeting the interdisciplinary evaluation requirements of this project.

Prior to the evaluation, all the evaluators underwent two rounds of standardized training: The first round focused on the five-dimensional evaluation indicator system, explaining the scoring rules, scoring boundaries, and deduction details for each dimension, and using examples to calibrate scoring, ensuring all evaluators had a consistent understanding of the indicators. The second round provided a basic explanation of the technical architecture and output logic of the three question answering schemes, clarifying that the entire evaluation process must strictly adhere to the double-blind principle, meaning that the evaluators cannot know the corresponding model scheme when scoring, and scoring is based solely on the question and answer themselves, minimizing the influence of subjective bias on the results.

The feedback from the participants indicated that the five-dimensional assessment framework developed in this study comprehensively covers the core assessment needs of smart city planning QA scenarios. The scoring rules for each dimension are clear and implementable, effectively reducing the subjectivity and arbitrariness of the assessment process. At the same time, the double-blind assessment design and standardized training process also ensure the fairness of the assessment process and the reliability of the results.

4.1.5. Experimental Procedure

Across all test questions, the three schemes strictly adhered to the single-variable principle throughout the comparative experiments. The model parameters and input content were held consistent across all three experimental schemes. The specific procedure was as follows:

KG-only scheme: The question was input, Cypher retrieval was executed on the Neo4j knowledge graph, and the raw structured results were returned as the output.

LLM-only scheme: The same question was directly submitted to ChatGLM4 to generate an answer, with the model temperature parameter uniformly set to 0.3.

Proposed integrated scheme: Following question classification, multi-source knowledge recall, and layered prompt injection, the system invoked ChatGLM4 (under the same configuration) to generate a response.

After summarizing and organizing all the question-and-answer outputs from the three models, the source information for the solutions was hidden to form an anonymous sample. This sample was then submitted to the review team for double-blind manual scoring based on the five evaluation dimensions, and a horizontal comparison was conducted to analyze the differences in the capabilities and optimization effects among the different frameworks.

4.2. Experimental Results

4.2.1. Answer Quality Comparison Across Frameworks

We compared three QA systems: a KG-only system, an LLM-only system, and our proposed KG-LLM integrated QA system. The results are presented in

Table 2. QA performance comparison across architectural systems.

Automatic QA System	Answer
	Question 1: How are “Physical Entity” and “Social Entity” in the “Entity Base Class” used to describe “Existing Buildings” and “Community Population Structure” in urban renewal?
Knowledge Graph Template Matching	Per GB/T 36332-2018, it is clearly defined that Physical Entity corresponds to Existing Buildings (as a physical space carrier) in urban renewal, and Social Entity corresponds to Community Population Structure (as a social activity subject). The two achieve the unity of physical space and social attributes through relationship mapping. At the same time, through three types of entities—land use, service element, and infrastructure—it can support the replacement of existing land functions and achieve cross-standard coordination of land use adjustment and public service allocation.
Large Language Model (ChatGLM4)	First, the core connotations of Physical Entity (material existence with objective reality, measurability, and stability) and Social Entity (dynamic systems composed of human interactions, relationships, and norms) are defined from philosophical and sociological perspectives. Then, corresponding to the Existing Buildings and Community Population Structures, the analytical logic and application value of the two in urban renewal are clarified. Finally, it is pointed out that urban renewal needs to achieve a synergistic balance between the transformation of physical space and the cultivation of social relationships.
Large Language Model Integrated with Knowledge Graph	Based on GB/T 36332-2018, this paper clarifies the correspondence between two types of entities and the core objects of urban renewal; it deduces the relationship logic between a Physical Entity and Existing Buildings and between a Social Entity and Community Population Structure, clarifying their application scenarios in spatial transformation and demand matching. It proposes the application logic of cross-entity collaboration, realizing the unification of land use adjustment and public service allocation through relationship mapping, and provides specific application cases. At the same time, it clarifies that the answers are based solely on the cited standards, defining the boundaries of the analysis.
Automatic QA System	Answer
	Question 2: A historic district renewal project needs to balance “cultural heritage preservation” and “smart transformation”. Based on the “Integration and Sharing” principle in “Top-Level Design Principles” and the “Place Creation” concept in “Community Living Circle Layout Requirements”, propose a technical pathway to resolve the conflict between “heritage preservation” and “equipment installation”, and specify which “Collaboration Base Class” process support is needed.
Knowledge Graph Template Matching	No applicable regulatory knowledge retrieved.
Large Language Model (ChatGLM4)	A three-stage technical path is proposed with “Integration and Sharing” as the core principle and “Place Creation” as the guide: ① Early collaboration stage: Carry out a two-dimensional analysis of style and demand, extract the core genes of the style, clarify the requirements for intelligent equipment, and identify the conflict points between the two. ② Technology integration stage: From the formation to the realization of an integrated design of equipment and style, based on the three dimensions of posture, function and technology, to reduce the impact of equipment installation on heritage through scenario-based, low-intrusion and shared design solutions. ③ Dynamic adaptation stage: Adopt flexible and modular installation solutions to cope with the uncertainty during the update process. At the same time, it is clear that four major collaborative base processes need to be supported, namely, the landscape assessment and demand alignment process, the collaborative design and scheme review process, the concealed engineering and installation acceptance process, and the operation and landscape dynamic monitoring process.
Large Language Model Integrated with Knowledge Graph	Based on the principle of “Integration and Sharing” and the requirements of “Place Creation”, four core technical paths are proposed: ① Hierarchical integration design: Divide the site into a ground feature layer, an equipment concealment layer, and a smart service layer, and achieve the visual isolation and functional integration of equipment and heritage through spatial layering. ② Modularization and reversibility technology: Use plug-and-play modular equipment and non-destructive installation technology, which can reduce damage to heritage and adapt to later adjustment needs. ③ Digital twin and virtual simulation: Build a digital twin model through Building Information Modeling (BIM) and Geographic Information System (GIS) to simulate the impact of equipment installation, and use Augmented Reality (AR) visualization to achieve multi-party collaborative plan review. ④ Sharing and reuse mechanism: Integrate equipment functions, reuse existing public facilities, and reduce repeated installations and new construction. At the same time, it is clear that four major collaboration base processes are needed to support it, namely, a cross-department collaboration process, public participation process, technical standards and specification process, and operation and iteration process. Once the applicable boundaries of the plan are defined, special areas, such as national cultural protection units, require additional special approval and stricter protection requirements.

Notes: The content of this table has been condensed; please refer to Appendix A for the full version.

. For Question 1—“How are ‘Physical Entity’ and ‘Social Entity’ in the ‘Entity Base Class’ used to describe ‘Existing Buildings’ and ‘Community Population Structure’ in urban renewal?”—the KG-only scheme produced content that conformed to current regulations, but its text presentation was rigid and lacked scenario extensibility. The LLM-only scheme excelled in language logic and fluency but was prone to generating fictitious content lacking a regulatory basis. In contrast, our proposed integrated scheme first centered on the regulatory text to clarify the official definitions and basic mappings, ensuring factual accuracy and authoritative credibility. It then leveraged the LLM’s logical and linguistic capabilities to elucidate the underlying connections between the Physical Entity and Existing Buildings, as well as between Social Entity and Community Population Structure, and further supplemented the analysis with practical application pathways, such as functional replacement of existing land. Finally, it explicitly demarcated the original regulatory text from reasonable inference, and distinctly marked the applicable boundaries of the knowledge.

For Question 2—“A historic district renewal project needs to balance ‘cultural heritage preservation’ and ‘smart transformation.’ Based on the ‘Integration and Sharing’ principle in ‘Top-Level Design Principles’ and the ‘Place Creation’ concept in ‘Community Living Circle Layout Requirements,’ propose a technical pathway to resolve the conflict between ‘heritage preservation’ and ‘equipment installation,’ and specify which ‘Collaboration Base Class’ process support is needed.”—the KG-only scheme failed to generate a valid answer due to incomplete coverage in the knowledge graph. The LLM-only scheme produced hallucinated content devoid of any regulatory basis. By contrast, the proposed integrated scheme performed a compliant and reasonable scenario inference grounded in regulatory principles and successfully generated a valid answer.

These results indicate that our proposed scheme preserved the precision and credibility of the knowledge graph while integrating the logical extension and generation capabilities of ChatGLM4. Relative to either standalone system, it demonstrated superior comprehensive QA performance and greater adaptability to boundary scenarios.

4.2.2. System Query Latency Analysis and Overall Scoring

The average scores of the three systems for the 20 questions are presented in

Table 3. Average Scores Across Five Dimensions.

Group	Accuracy	Completeness	Fluency	Credibility	Scenario Relevance	Overall Average
Control 2 (KG only)	5.0	2.1	2.0	5.0	2.8	3.38
Control 1 (LLM only)	2.3	4.9	4.8	1.8	4.0	3.56
Experimental Group (proposed)	5.0	4.6	4.5	4.9	4.8	4.76

Note: The KG-only scheme returns “no relevant regulatory knowledge” on most inferential advisory questions, yielding lower scores for completeness, fluency, and scenario relevance. The correlation coefficient of the manual scoring results is 0.82, which is higher than the excellent reliability threshold of 0.8. This indicates that the scoring by the reviewers is consistent and the data for this manual evaluation has statistical reliability. The subsequent comparative analysis of scores in each dimension is effective.

.

In a unified experimental environment, based on 20 test samples, the end-to-end response latency was statistically analyzed: the KG-only scheme performed structured knowledge base retrieval through Cypher statements, without involving the large model’s text generation process, with an average single query time of 0.08 s; the LLM-only scheme relied on the model to complete the entire natural language generation process, with an average query time of 3.2 s; the KG-LLM fusion scheme presented in this paper integrated the entire process of knowledge graph retrieval, multi-source legal provision aggregation, hierarchical prompting guidance, and model reasoning, with an average response time of 3.3 s. From the perspective of latency performance, the fusion solution had only a slight time loss compared to the pure LLM baseline, but it significantly optimized the factual compliance of the answer by leveraging the structured knowledge constraints. The overall response efficiency can meet the actual implementation requirements of online consultation in the planning industry.

The overall scoring results are summarized as follows:

KG-only scheme: For deterministic questions already encompassed by the regulatory standards, this system achieved full marks in factual accuracy and credibility. Its output adhered strictly to national standard provisions, exhibiting no fabrications or hallucinations. However, its answers were mechanically structured, highly fragmented, and lacked both logical coherence and natural language expression. For inferential or comprehensive questions falling outside the coverage of the knowledge graph, it was unable to provide valid and complete responses. Consequently, its overall average score was 3.38.

LLM-only scheme (ChatGLM4): This system excelled in breadth of coverage, language fluency, and scenario adaptability. Its output was content-complete and logically coherent, and it actively aligned with the practical needs of smart city planning and urban renewal. However, owing to the absence of constraints from authoritative regulatory standards, it demonstrated a pronounced weakness in factual accuracy, often generating subjective fabrications that deviated from regulatory principles. Its information sources were vague, its traceability was poor, and its overall credibility was insufficient. Its overall average score was 3.56.

Proposed KG-LLM scheme: This system seamlessly integrated the authoritative precision of the knowledge graph with the flexible generation capability of the large language model. It substantially enhanced both answer accuracy and credibility, while also producing logically complete and professionally expressed explanations supported by traceable regulatory references. Even for complex planning problems beyond the direct scope of the knowledge graph, the system was capable of conducting compliant and reasonable reasoning grounded in underlying regulatory principles, thereby effectively suppressing unfounded hallucinations. Simultaneously, it retained strong language fluency and practical scenario adaptability. Its overall performance surpassed both baseline systems across all dimensions, achieving a final score of 4.76.

5. Discussion

The experimental results demonstrate that the proposed integrated LLM–knowledge graph QA system for smart city planning significantly outperforms both baseline systems in overall performance, effectively mitigating the inherent limitations of each single framework. This advantage derives primarily from three complementary strengths. First, by simultaneously injecting rigid constraints from the structured knowledge graph and a standard-level vector knowledge base, the system preemptively suppresses hallucinations, substantially improving answer accuracy, credibility, and regulatory compliance. Meanwhile, through centrality and clustering quantitative analyses, it can be seen that the clustering results of the knowledge graph in this field are highly consistent with the four-layer ontology. The hub nodes are concentrated in the legal entries, which ensures the authority of the search content from a structural level. It also provides the underlying structural foundation for the compliance of the integrated system answers, making up for the shortcomings of the general large model without structured knowledge constraints. Second, relative to a standalone general-purpose LLM, a pure search engine, or a knowledge graph-only approach, this system is more targeted in its domain, more rigorous in the expression of professional terms, and has significantly fewer factual deviations. It is capable of providing planning professionals and grassroots managers with accurate, compliant, and traceable knowledge inquiries and decision-making assistance for analysis and deduction. Third, the three-tier collaborative paradigm established in this work—comprising “precise graph retrieval, vector semantic recall, and LLM-fused generation”—can not only directly empower intelligent knowledge services for smart city planning sub-scenarios, but also serve as a transferable and reusable standardized reference framework for developing intelligent QA tools in other vertical domains, such as territorial spatial planning and urban–rural governance.

Furthermore, the system relies on a domain-specific knowledge base constructed from homogeneous sources, thereby enabling structured storage, visual presentation, and end-to-end reasoning of planning regulatory knowledge. This allows practitioners to rapidly retrieve the provisions and trace the constraint logic. Across the three typical urban renewal QA tasks—factual definition, relational constraint, and inferential advisory—each output element is labeled with its regulatory source, and the entire knowledge derivation process is traceable, ensuring the compliance and credibility of decisions in professional scenarios. This helps reduce human bias in planning decisions, facilitates the standardization of planning review processes, and has good engineering implementation value.

From the perspective of its practical implementation in urban renewal, planners in frontline operations, such as urban renewal and community infrastructure construction, generally face pain points, including scattered regulatory items, cumbersome cross-standard comparisons, and time-consuming manual reviews. This system can deeply assist practitioners in the preparation of planning schemes. During the scheme conception stage, it can quickly retrieve the relevant national standard constraints, such as supporting facilities, disaster prevention, and land use, helping design content that aligns with current regulations. During the scheme verification stage, it automatically compares texts with legal provisions, proactively identifying non-compliant design content. This system can be applied to routine scenarios, such as planning scheme compliance verification, community living circle infrastructure acceptance, and renewal project pre-approval. Practitioners can significantly reduce the time spent on data retrieval and clause verification, efficiently promoting scheme implementation and empowering the refined construction of urban renewal projects. From an industry development perspective, the practical implementation path of this study provides a lightweight implementation example for the digital transformation of smart planning, aligning with the industry development trend of refined urban construction.

6. Limitations

Nevertheless, this study is subject to certain limitations. First, the generalization capability of template matching is constrained. The current system relies primarily on precise graph retrieval and rule-based constraints for factual definition and strong relational questions. While the accuracy remains controllable, the system’s ability to generalize to complex long sentences, non-standard paraphrasing, and cross-domain compound questions is limited. Second, the coverage of the knowledge graph remains insufficient. The knowledge graph constructed in this study draws upon only four industry standards, yielding a restricted number of entities and relations. It does not explore the emerging planning concepts and implicit cross-standard associations sufficiently.

7. Conclusions and Application Prospects

To address the above shortcomings, future research can improve the system in the following ways: First, introduce deep domain-specific semantic matching models to supersede traditional keyword-triggered logic and extend support for multi-hop reasoning and aggregation-based complex questions [26], thereby comprehensively improving the system’s capacity to address complex tasks. Second, multi-level industry standards, local technical guidelines, and well-established practical cases can be gradually introduced. By leveraging automated knowledge extraction technology, entity attributes can be enriched, and a larger and more comprehensive domain knowledge base covering a wider range of fields can be constructed.

The integrated LLM–knowledge graph QA system for smart city planning in this study can effectively empower planning practices, assist practitioners in completing tasks such as planning scheme preparation and compliance verification, effectively reduce the cost of reviewing regulations and comparing clauses, improve the efficiency of project design and review, help urban renewal projects to be implemented in a refined manner, and provide a lightweight technical reference for the digital transformation of the smart planning industry.