From Heritage Building Information Modelling Towards an ‘Echo-Based’ Heritage Digital Twin

Abstract

Since the late 2000s, numerous studies have focused on the application of Heritage Building Information Modelling (HBIM) processes and technologies for the documentation of the historic built environment. Many of these studies have focused on the use of BIM software tools to generate intelligent 3D models using information gathered from a range of data capture techniques including laser scanning and photogrammetry. While this approach effectively preserves existing or partially extant heritage, it faces limitations in reconstructing lost or poorly documented structures. The aim of this study is to develop a novel approach to complement the existing tangible-based HBIM methods, towards an ‘Echo-based’ Heritage Digital Twin (EH-DT) an early-stage digital representation that leverages intangible, memory-based oral descriptions (or echoes) and AI text-to-image generation techniques. The overall methodology for the research presented in this paper proposes a three-phase framework. Phase 1: engineering a standardised heritage prompt template, Phase 2: creation of the Architectural Heritage Transformer, and Phase 3: implementing an AI text-to-image generation toolkit. Within these phases, intangible data, including collective memories (or oral histories) of people who had first-hand experience with the building, provide ‘echoes’ of past form. These can then be converted using a novel ‘Architectural Heritage Transformer’ (AHT), which converts plain language descriptions into architectural terminology through a generated taxonomy. The output of the AHT forms input for a pre-created standardised heritage prompt template for use in AI diffusion models. While the current EH-DT framework focuses on producing 2D visual representations, it lays the foundation for potential future integration with HBIM models or digital twin systems. However, the reliance on generative AI introduces potential risks of inaccuracies due to speculative outputs, necessitating rigorous validation and iterative refinement to ensure historical and architectural credibility. The findings indicate the potential of AI to extend the current HBIM paradigm by generating images of ‘lost’ heritage buildings, which can then be used to enhance and augment the more ‘traditional’ HBIM process.

1. Introduction

Architectural heritage is a cornerstone of cultural identity and collective memory, embodying the historical, social, and artistic narratives of communities [1]. Preserving this heritage presents significant challenges, particularly in the face of physical degradation and the absence of comprehensive documentation or archival records [2]. Traditional conservation approaches focus on the tangible aspects of heritage, such as buildings and monuments, often overlooking the critical intangible elements, including oral histories, functional uses, and social rituals [3].

The integration of both tangible and intangible heritage is essential to developing holistic preservation strategies, particularly in cases where physical structures are partially or entirely lost due to war, natural disasters, or neglect [4]. The loss of these structures necessitates alternative approaches that incorporate oral histories, photographs, and written narratives to recreate and preserve their memory [5]. This complexity has led to the exploration of new methodologies that embed non-physical data into conservation frameworks, extending the scope of preservation beyond material artefacts [6].

Ontological frameworks have emerged as a promising solution to integrate diverse heritage data. These frameworks provide structured and formalised representations of knowledge, facilitating the organisation, sharing, and reuse of data across different platforms [7]. Ontologies not only standardise data but also ensure the seamless integration of tangible and intangible heritage elements within a unified semantic network [8,9]. As a result, ontologies enhance interoperability, supporting collaborative conservation efforts across disciplines and organisations [10]. For example, the CIDOC Conceptual Reference Model (CRM), an ISO standard since 2006, illustrates this approach by representing both physical and cultural narratives, contributing to comprehensive heritage management [11]. Despite the advancements in ontological frameworks, challenges persist in the application of these systems to heritage conservation, particularly in accommodating the diverse cultural contexts and specific needs of different heritage domains [12]. The existing models often prioritise physical documentation, leaving gaps in representing intangible elements crucial to understanding and preserving heritage [13]. The integration of ontologies with Building Information Modelling (BIM) has been identified as a critical step towards addressing these gaps, creating comprehensive 3D representations enriched with semantic data [14]. Heritage Building Information Modelling (HBIM) extends BIM practices to heritage conservation, enabling the detailed documentation and management of historic structures [15]. HBIM has proven invaluable for preserving architectural heritage by generating accurate 3D models and supporting restoration efforts [16]. While HBIM excels in representing existing heritage buildings through data collected from laser scanning, photogrammetry, and measured surveys, its application to lost or partially destroyed heritage assets presents unique challenges [4]. In the absence of physical remains, reconstruction efforts rely heavily on archival photographs, sketches, oral testimonies, and historical narratives [17]. These multidisciplinary approaches have been employed in notable projects of lost heritage [18,19], yet the integration of intangible data into HBIM workflows often requires manual interpretation by experts, resulting in inconsistencies and variability across reconstructions. The need for alternative, automated approaches to facilitate the inclusion of intangible heritage in digital models has driven recent exploration into AI-driven solutions that leverage community memories and descriptive narratives. AI-driven approaches have the potential to enhance HBIM by incorporating intangible data, thus addressing limitations in reconstructing lost heritage.

Building on this momentum, the Echo-based Heritage Digital Twin (EH-DT) framework is proposed. The EH-DT integrates AI-driven techniques, such as text-to-image generation, with HBIM and ontologies to create initial visual representations of heritage buildings, particularly those that no longer physically exist. This novel framework leverages intangible data sources, such as oral histories, to enrich digital models, providing a more comprehensive representation of heritage assets.

The EH-DT represents an early stage in digital twin development, aligning with pre-digital twin maturity levels by generating 2D visual reconstructions from oral narratives. The framework emphasises iterative refinement, engaging former occupants or stakeholders to validate and enhance AI-generated images until a collective consensus is achieved. By embedding AI capabilities and digital twin principles, the EH-DT complements traditional HBIM methodologies, positioning itself as an innovative solution for the preservation and reconstruction of lost heritage.

However, it is essential to recognise the limitations of AI-generated imagery within this framework. Generative AI tools often extrapolate details beyond the input data, which can compromise historical and architectural accuracy if left unchecked. To mitigate this, the EH-DT framework incorporates iterative validation processes involving stakeholders and experts to refine outputs and address inaccuracies. Despite these precautionary measures, AI-generated models should be considered supplementary tools rather than definitive reconstructions.

The research presented in this paper introduces a structured three-phase framework for developing the EH-DT, extending beyond tangible-focused HBIM workflows:

Phase 1—Engineering a Standardised Heritage Prompt Template (SHePT).

A template is developed to standardise the process of converting oral histories into structured prompts.

Phase 2—Creation of the Architectural Heritage Transformer (AHT).

The AHT ontology is designed to translate plain language oral histories into formal architectural terminology. This phase ensures that the data fed into the AI generation tools accurately reflect architectural features, materials, and historical contexts, bridging the gap between community-driven narratives and technical requirements.

Phase 3—Implementing AI Text-to-Image Generation.

The AHT-generated outputs are processed through AI text-to-image tools (such as DALL-E and Stable Diffusion), creating initial visual representations of the lost heritage buildings. This phase leverages OWLready2 Python scripts to automate the generation of prompts, pulling architectural elements directly from the ontology and ensuring accuracy in the AI-generated imagery.

To evaluate the EH-DT framework, a pilot case study was conducted on the Church of St Michael, Alberbury with Cardeston, a 12th-century structure in Shropshire, England. This pilot assessed the framework’s ability to generate visual reconstructions using oral histories collected from local residents, architects, and archival records. The results demonstrated the potential of the EH-DT to complement traditional HBIM approaches, offering a pathway to reconstruct lost or damaged heritage buildings through community-driven intangible data sources.

2. Related Works

This section reviews the current advancements in heritage conservation, focusing on the role of architectural heritage (AH) and ontologies, the integration of intangible data into HBIM workflows, and the emerging use of AI text-to-image generation tools. It also describes the levels of maturity within digital twin frameworks, outlining their development stages and positioning the EH-DT framework within this progression.

2.1. Architectural Heritage (AH) and Ontologies

Architectural heritage includes buildings and artefacts that reflect the history, culture, and experiences of the people associated with them [20]. These assets are often globally recognised for their tangible forms and physical characteristics, such as age, aesthetics, and authenticity. Heritage assets include a diverse array of buildings, monuments, sites, areas, and landscapes. Their significance and vulnerability necessitate careful consideration during conservation and restoration efforts [21]. Numerous researchers [22,23] highlight that the intangible aspects of heritage, such as oral histories, social rituals, and building functions, are equally important to preservation efforts. However, Ref. [3] argues that form often receives more value than function, leading to the under-representation of vital intangible elements. This bias is reflected in traditional conservation methodologies, which predominantly address physical deterioration while neglecting the intangible narratives and rituals associated with heritage buildings [24].

To overcome these limitations, one strategy is the development of ontological frameworks, which provide structured methods for representing both tangible and intangible heritage elements, enabling a holistic understanding of architectural assets [7]. Ontologies serve as critical tools for organising, standardising, and managing cultural heritage data, bridging the gap between geometric information and historical narratives while promoting interoperability across disciplines and institutions [25]. One of the key benefits of using ontologies is their ability to provide a uniform representation of both tangible and intangible heritage data within a single semantic network. This includes the seamless integration of concepts, relationships, functions, rules, and constraints, making it easier to share and understand information across the scientific community [8,9].

By providing a structured representation of knowledge within a specific domain, ontologies play a crucial role in data processing, logical reasoning, sharing of information and enabling the re-using of knowledge [9]. This enhances interoperability between individuals and organisations [10]. For instance, the CIDOC CRM ontology was initially developed to manage museum artefacts but has since evolved into a core reference model for cultural heritage [11]. Its ability to represent complex cultural heritage information, such as spatial and temporal concepts, makes it invaluable for managing both physical structures and the associated cultural narratives [8]. Other domain-specific ontologies have been developed to address specialised aspects of heritage conservation. In Algeria, a knowledge ontology was created to capture and interpret the mixed architecture of the city of Blida, helping to preserve both its tangible and intangible qualities [26]. In China, an ontology was developed to document and conserve Chinese architectural heritage using standardised classification methods, which resulted in the generation of comprehensive 3D models and a library of historical building components [27]. Other examples include the Monument Damage Information System (MONDIS), designed to document and diagnose damaged historical structures, and the Architecture Metadata Object Schema (ARMOS), which focuses on cataloguing the formal aspects of architectural design [28,29]. In addition, the Geneva CityGML ontology supports the creation of 3D models of historical buildings, allowing for enhanced digital preservation [30].

Innovative uses of ontologies also extend to intangible heritage. For example, an ontology-based system was designed to model cultural contact within Southern Chinese martial arts using Semantic Web standards to extract knowledge from archival materials [31]. Ontologies also play a crucial role in risk management, as demonstrated by a context-aware risk management model developed for HBIM and Virtual Reality (VR), which allows for the efficient sharing of risk-related data between conservators and heritage managers [32].

Despite their widespread utility, the application of ontologies in architectural heritage conservation is not without challenges. The diversity of cultural contexts and the specific needs of different heritage domains can limit the broader application of existing models [12]. Additionally, while traditional metadata standards and 3D modelling tools like BIM provide rich geometric data, they often fall short of capturing the full historical and cultural context of heritage structures [13]. To address these challenges, the integration of ontological frameworks with BIM has been identified as a promising solution for managing both semantic and geometric data in a holistic manner [14].

However, the existing ontologies tend to focus on physical structures, often overlooking fragmented or community-driven narratives essential for reconstructing lost heritage. The AHT ontology addresses this by translating plain language responses and oral histories into structured architectural terminology, bridging the gap between technical data and intangible heritage. Phase 2 of the EH-DT framework methodology will provide a detailed description of the AHT ontology and its development.

2.2. HBIM and Integrating Intangible Data for Lost Heritage

Building Information Modelling (BIM) is a widely adopted modelling technology and process that facilitates the creation, communication, and analysis of building models [33]. Originally developed for new-built assets, BIM provides a 3D digital representation enriched with functional data, supporting decision-making across the lifecycle of a project [34]. Beyond geometric modelling, BIM incorporates additional dimensions—4D (time), 5D (cost), 6D (sustainability), and 7D (maintenance)—to enable comprehensive asset management [35].

When applied to existing or historic buildings, BIM evolves into Heritage Building Information Modelling. HBIM adapts BIM methodologies to preserve, document, and manage architectural heritage by integrating historical, archaeological, and architectural data [15,16]. This facilitates accurate 3D models of heritage buildings, assisting in restoration and conservation efforts [36]. HBIM extends BIM’s utility beyond contemporary structures, addressing the complexities of irregular forms, deformations, and uncertainties inherent in historical assets [37].

The HBIM workflow typically involves three critical stages: (1) data collection, (2) data processing, and (3) data fusion [38]. Each stage plays a vital role in generating comprehensive models that represent both the physical characteristics and cultural essence of heritage buildings.

The first stage of the HBIM process involves collecting diverse datasets that contribute to a comprehensive representation of heritage buildings. This includes geometric data obtained through laser scanning, photogrammetry, and manual surveys, as well as archaeological and historical data derived from archival records, past drawings, and photographs [39]. Additionally, material and pathology data are gathered to assess the building’s condition and identify signs of deterioration [40]. Performance data, such as energy efficiency, moisture levels, and indoor environmental quality, are also incorporated to provide insights into the building’s functional health [41]. Beyond tangible aspects, intangible data—such as oral histories, rituals, and the functional use of spaces—play a crucial role in preserving the cultural and social memory of the structure [42]. By integrating these diverse data sources, the HBIM model not only captures the physical form of a building but also reflects its broader cultural significance and historical context [6].

Once the data are collected, the second stage—data processing—focuses on transforming these raw datasets into intelligent 3D models. The method employed depends on the condition of the heritage building, with two primary approaches being used. For existing heritage structures, the scan-to-BIM process leverages geometric data captured through laser scanning or photogrammetry to produce digital models [43]. This technique generates point clouds that are segmented and converted into HBIM models, effectively documenting the current state of standing heritage buildings. The scan-to-BIM approach not only facilitates structural assessments but also aids in future conservation planning by providing accurate, detailed representations of the building’s form and condition [44].

When heritage assets are partially or fully destroyed, or if physical remains are insufficient for traditional surveys, the lost heritage approach is applied [45,46]. This method relies on archival materials such as drawings, photographs, and sketches to inform digital reconstructions. Additionally, oral histories and local narratives gathered from former occupants, residents, or stakeholders provide crucial qualitative data that fill gaps left by incomplete physical records [39,47]. Although oral data introduces uncertainty, practitioners address this by applying colour coding to distinguish reconstructed elements based on their source. Photographic evidence is typically visualised in neutral tones, while elements derived from oral histories or uncertain sources are rendered in distinct colours to indicate higher speculation [48]. This approach was exemplified in the reconstruction of Ireland’s Magdalene Laundries, where sections derived from survivor testimonies were highlighted in bright blue, adhering to the London Charter’s guidelines for representing uncertainty [49].

The lost heritage approach integrates diverse data sources, including archival photographs, newspaper illustrations, and artistic renderings, to create comprehensive reconstructions. When visual records are scarce, community engagement through oral testimonies becomes invaluable, allowing practitioners to capture recollections of architectural features, materials, and spatial configurations. This method has been successfully applied in cases such as the Altai Mission Churches in Russia, where oral accounts supplemented incomplete surveys to recreate digital models of mission churches [50]. Similarly, the McCarren Pool in New York City was reconstructed using memories from over 20 community members, reflecting not only the architectural layout but also the social and sensory experiences tied to the space [51]. Vernacular documentation, where local artisans and residents contribute on-site sketches and personal knowledge of traditional building techniques, further enriches reconstructions in areas lacking formal records. This participatory approach played a key role in rebuilding Acehnese heritage houses, translating community knowledge into digital models [5]. Crowdsourced data from online archives and social media platforms also serve as valuable resources for heritage reconstruction, as demonstrated by the digital recreation of the Al-Hadba Minaret in Mosul using tourist photographs and videos [46].

In the final stage of the HBIM process, data fusion combines multiple data sources to create a comprehensive 3D HBIM model including detailed information that is required for historical conservation and management purposes [38]. The resulting 3D models and accompanying documents attain a variety of applications, such as drawing, 3D printing, virtual tours, and physical reconstruction [46].

2.3. Exploring AI Text-to-Image Generation Tools in HBIM

AI text-to-image generation has emerged as a powerful tool with potential applications in various fields, including architecture [52,53,54,55]. These tools are transforming how architects approach concept design, enabling them to quickly visualise ideas from simple textual inputs [56]. Recent advancements in AI text-to-image generation tools, such as MidJourney, Stable Diffusion, and DALL-E, have gained popularity for their application in the early stages of architectural design ideation and concept generation [52].

In architectural design, these AI tools allow for iterative concept generation. Designers input textual prompts, which AI interprets to produce visual outputs that evolve through refinements and feedback [57]. For instance, MidJourney excels at creative iterations, while DALL-E’s capabilities extend to urban planning and spatial design tasks, albeit with limitations in capturing architectural specificity and complex geometric details [58]. Although these tools facilitate the generation of design inspirations, manual refinement is often required to align outputs with architectural precision [59].

In the realm of heritage conservation and HBIM, AI-generated imagery holds great potential, particularly in addressing challenges associated with reconstructing lost or partially destroyed buildings. Current HBIM practices for lost heritage rely heavily on historical archives, sketches, and oral histories to model heritage sites [48]. AI text-to-image generation offers a complementary avenue by translating intangible data into visual reconstructions, filling in gaps left by incomplete records or fragmented datasets. For example, DALL-E’s ‘outpainting’ feature has been applied to restore ancient mosaics and cultural artefacts by expanding existing imagery based on limited visual references [60].

Despite its potential, the use of AI in architectural heritage is still in its infancy. Studies like [61] explore AI-generated imagery in architectural education, enhancing engagement and fostering creative exploration of historical structures. However, the direct integration of AI-generated images into heritage workflows—specifically within HBIM for lost heritage—remains largely untapped. The existing applications focus on conceptual design, with limited emphasis on the semantic accuracy and historical authenticity essential for heritage models. This research seeks to bridge this gap by exploring the use of AI-based tools to translate oral histories and fragmented descriptions into visuals of lost heritage buildings. The third phase of the EH-DT framework leverages AI text-to-image technology to complement traditional HBIM workflows, offering a novel method for reconstructing intangible aspects of heritage.

2.4. Digital Twin Maturity Levels and the Positioning of the EH-DT

The concept of digital twin maturity is evolving across industries, with various frameworks attempting to classify the sophistication and capabilities of digital twins. These models generally range from basic digital representations to autonomous and intelligent systems capable of real-time interaction and decision-making [62,63]. Despite the diverse terminologies and classifications, most studies agree on a progression that begins with static, descriptive models and advances towards fully autonomous, cognitive twins [64]. In the context of heritage conservation, Heritage Digital Twins (HDTs) are becoming essential for preserving, interpreting, and managing historical assets. Studies by [65,66] indicate that HDTs have the potential to revolutionise the heritage sector by supporting the documentation, monitoring, and management of historical structures. However, unlike the digital twins used in modern industries that primarily rely on sensor data and operational inputs, heritage-focused models often contend with fragmented records, intangible heritage, and the absence of real-time monitoring. This distinction places many heritage-focused frameworks, including the EH-DT, at the lower end of the maturity spectrum.

The EH-DT aligns with the pre-digital twin level (Level 1), which is typically characterised by basic digital models, 2D imagery, or CAD representations that visually describe an asset [64,67]. At this stage, digital twins answer the question, “what happened?”, by providing a historical or visual account without incorporating dynamic or real-time data. For the EH-DT, this is represented by AI-generated images derived from oral histories, archival materials, and community recollections. These images serve as foundational representations that reflect the tangible and intangible characteristics of heritage assets, even in cases where physical remains are limited or entirely absent. This approach is echoed by Ref. [68], who argue that the initial stages of HDT development often involve generating static models based on archival surveys and photogrammetric data.

Unlike standard Level 1 digital twins, which are often geometric in nature, the EH-DT introduces intangible heritage data early in the process by keeping a record of memories, rituals, and social functions along with the visual output. This approach allows for the preservation of cultural narratives alongside architectural features, setting the EH-DT apart from conventional descriptive models that focus solely on form and structure. The authors of [69] similarly emphasise the importance of linking digital models to cultural and functional aspects of historical buildings to ensure that digital twins reflect the lived experience and social significance of heritage structures.

Level 2 (Digital Shadow) represents the next stage, where digital twins aggregate multiple data sources to enrich the model, typically involving one-way automated data flows between the physical asset and its digital counterpart [70]. This level seeks to answer, “why did it happen?”, reflecting a deeper engagement with data quality and completeness.

Level 3 (IoT Digital Twin) introduces real-time monitoring through sensor integration, addressing “what will happen?” by predicting future performance based on live data inputs [64,71]. This predictive capability is a hallmark of the modern digital twins used in smart infrastructure and manufacturing. Research by [72,73] demonstrates early efforts to apply sensor technologies to heritage buildings, enabling performance monitoring and preventative maintenance.

Further along the spectrum, Level 4 (Prescriptive Digital Twin) leverages AI learning models to suggest interventions, while Level 5 (Cognitive Digital Twin) introduces self-optimisation and adaptive capabilities driven by continuous feedback [63,74]. The authors of [66] propose that AI-driven digital twins could enhance restoration efforts by identifying patterns in historical data, but such applications remain largely theoretical in the heritage domain. These advanced stages remain largely out of scope for heritage-driven frameworks like the EH-DT, where the emphasis lies on memory preservation and architectural reconstruction rather than automation or performance optimisation.

By aligning the EH-DT with the pre-digital twin category, this framework highlights the importance of foundational representations in heritage conservation. The focus on capturing intangible heritage and translating oral histories into AI-generated imagery reflects a novel approach that situates the EH-DT at the intersection of historical documentation and digital innovation. This positioning underscores the role of the EH-DT as an early-stage digital twin model, contributing to the preservation of cultural heritage while laying the groundwork for future advancements [65,68].

3. Method and Materials

The overall method used in developing the proposed EH-DT is presented in this section (Figure 1). The input data comprise raw oral histories of the previous occupants or those who had once first-hand experiences with the ‘lost’ heritage building. These input data are then used within a structured framework process that can integrate oral histories and descriptions that provide an ‘echo’ of the past for use with emerging AI-based tools.

Figure 1. Proposed framework process.

To ensure the creation of a high-quality and useful oral history, the process should start with a plan to gather a comprehensive and detailed amount of information within the focus area [75]. To ensure the raw data gather the required information, it is essential to develop questions that will be used to extract precise details from individuals, which can then be fed into a diffusion AI text-to-image prompt. The structure of questions can be formulated by identifying the specific AI prompt layout (template) that gives optimal results in generating heritage buildings that present a high similarity to lost heritage assets.

Oral histories can serve as textual descriptions that are needed for AI-based tools. However, in order to ensure that AI prompts use the correct architectural descriptions and vernacular, an approach is needed to transform plain language descriptions into specific information that can then be fed into an AI Text-to-image tool. Directly inputting oral histories or spoken language into an AI text-to-image generation will produce erroneous results, as these tools require precise and structured architectural terminology to enhance the relevancy and accuracy of the generated images [52,76,77].

4. Engineering a Standardised Heritage Prompt Template (SHePT) (Phase 1)

AI prompt templates are ‘prompts with slots’ that enable user customisation [78], providing a structured way to describe the subject, form, and content of the prompt [79]. The objective of this phase is to develop a structured template that generates images of heritage buildings as accurately as possible because it will need to be able to provide the form and aesthetic of the building along with specific features that may be pertinent to the historic period of use. In order to achieve this, the process of creating or engineering a prompt template for historic buildings was undertaken by aligning with Wallas’ four-stage model of the creative process [80]: preparation, incubation, illumination, and verification [81]. The creation of the template was seen as a creative endeavour, and so the four-stage model provided a solid philosophical underpinning.

Within these stages, and in order to develop a standardised heritage prompt template (SHePT), a case study approach was implemented involving multiple historic buildings. The Heritage at Risk (HAR) register in the UK is developed and maintained by Historic England to identify sites that have the potential to be lost due to decay or neglect due to inappropriate development [82]. The HAR register was chosen for this study due to its publicly available comprehensive listings of heritage buildings that are at risk due to various factors. Using the register, a search was undertaken for Grade II listed buildings in different parts of England, which were deemed to be in poor condition with a slow decay. In addition, further selection was undertaken such that the buildings had elements of Victorian-era construction as the wider scope of this study is to specifically focus on buildings of this era. In total, 13 buildings were selected for the initial development of the SHePT (Figure 2).

Figure 2. Development of a standardised heritage prompt template.

4.1. Preparation

The objective of the preparation stage is to gather the necessary resources and information to begin the prompt template creation; importantly, preparation relies on acquiring domain-specific knowledge [80]. This is completed through the following:

(a)

Data Collection: As discussed above, the primary source of data for this part of this study were descriptions and images of buildings selected from the Heritage at Risk (HAR) register in England.

(b)

Tool Selection: Appropriate AI text-to-image generation tools are evaluated for their suitability in generating heritage buildings. By reviewing currently available tools, DALL·E 2, Stable Diffusion, Disco Diffusion, Adobe Firefly, and Imagine AI Art were considered for this study. This study excluded Midjourney and Mnml due to their more limited availability and reliance on non-textual prompts, respectively. The final selection of DALL·E 2, Stable Diffusion, and Adobe Firefly was based on their potential application in architectural applications [59,83,84].

4.2. Incubation

In its literal sense, the incubation stage involves pausing while working on the creative problem and linking new information to existing knowledge or switching focus to other topics, which can later lead to innovative insights and creative solutions [80]. This stage involves reflecting on the collected text-based descriptive data from the HAR along with the selected tools to reflect on the potential formulations of prompts. At this stage, the information from the HAR register was organised and categorised into a tabulated outline for clarity, and from this, two types of prompt formulations were identified for testing.

(a)

Direct Prompt Method: this method directly uses descriptions of heritage buildings from the HAR register as an initial prompt to ensure an accurate and appropriate description of the building.

(b)

Reverse Engineering Method: this method is based on the visual captioning process, where a descriptive sentence is generated for an image [85]. This study used ChatGPT-4 to convert images of heritage buildings from the HAR register into textual prompts. These generated prompts were then used in the three AI tools to produce images.

This process was used to analyse the potential differences between a human-written and an AI-written prompt to understand potential nuances that AI tools may see when generating future images.

4.3. Illumination

Illumination is the stage where insights are acquired by applying the best possible ways or ideas to solve the problem [80]. To identify the best possible AI solutions, images of the 13 selected buildings were generated using the direct prompt method and reverse engineering method. Table 1 provides an exemplar of a comparative table developed for each of the 13 selected buildings, which includes the prompts used for both methods, the original image of the heritage building as listed in the HAR register, an image of the building generated using the direct prompt method, and, lastly, an image of the building generated using the reverse engineering prompt method. Similar tables were created for all 13 buildings.

Table 1. Exemplar comparative table of AI diffusion results from alternate prompts.

Building Name: Church of St Mary, Fretherne with Saul—Stroud
Original Photographs from Historic England	Source: Historic England [21]
Description from HAR Direct Prompt	Large mid-Victorian parish church in an isolated site. Ornate in the decorated style of sandstone with limestone dressings. There has been a loss of stonework with many falls of carved features.
Reverse Engineering Prompt	Capture a realistic photo of an intricate, Victorian-style church surrounded by a lush graveyard. The church’s detailed stonework, pointed spires, and ornate windows stand out against a blue sky. This photograph aims to emphasise the church’s architectural beauty and its serene, historical setting within the cemetery, showcasing the harmony between human craftsmanship and nature.
AI-Generated Images
AI Text-to-Image Tool Used	HAR Description	Reverse Engineering Prompt
DALL·E 2
Stable Diffusion
Adobe Firefly

4.4. Verification

Verification is the stage where the results are evaluated and potentially refined into a final solution. This stage provides a ‘book-end’ throughout the stages [80]. For this study, the verification consisted of three phases as described below.

• Testing and review of generated images.

A survey method was employed to compare each set of AI-generated images to the actual photographic images of the 13 buildings selected from the HAR register. The survey involved a sample size of 16 participants consisting of both subject experts (including architects, built environment professionals, and architectural students) and non-subject experts. Studies involving human evaluation of AI-generated images, such as [86], demonstrate that even relatively small groups can effectively assess the quality and similarity of generated images. The participants were shown the original image of the building followed by six AI-generated images from each diffusion model of the same building, as illustrated in Table 1. Understanding that the images would not depict perfect replicas of the physical building, the participants were asked to rate the images based on their similarity to the original photographs on a scale of 1 to 4, with 4 being the most similar. The respondents were asked to look at a range of elements including architectural styles, materials, features, and geometric form.

The scores were aggregated to understand which of the software tools produced the most ‘similar’ results and also understand if either of the two used descriptions provided more representative examples of the test building (Table 2 and Table 3). An initial evaluation of the results highlighted that most of the respondents found that more similar results came from the use of the DALL-E tool. A deeper analysis of the responses highlighted that when using any of the AI diffusion tools selected, the most ‘similar’ images were created from the direct prompt method, namely, using the original description of the building as stated in the HAR register.

Table 2. Analysis of a successful prompt.

Prompt	Subject	Form	Content
Building 13: Large mid-Victorian parish church in an isolated site. Ornate in the decorated style of sandstone with limestone dressings. There has been a loss of stonework with many falls of carved features.	Type of building—Architectural style/year—setting	Architectural elements—Architectural elements materials	Current state deterioration

Table 3. Mapping of SHePT slots into existing ontologies.

SHePT Slot	Relevant Ontologies	Descriptions
Number of storeys	BOT	Provides core topological concepts of buildings, including number of storeys
Type of building	BOT	BOT describes building types and structures
Style/age of building	BHP	Describes the architectural style and historical periods of buildings
Material/colour of building	ConTax	Covers construction materials and their properties
Specific architectural elements	BOT, ConTax	Details on building elements
Material of elements	BOT, BHP	Both BOT and BHP describe materials, with BHP offering heritage-specific details
Context/environment	BOT, ConTax	Provides context such as site and environmental zones
Architect	BHP	Information about architects related to the building
Restoration details	CTO, BHP	Describes restoration tasks and details
Signs of deterioration	MWV-D, DOT	Describes observable damages and their types
Current use or occupancy	BHP	Provides details on the current function of the building
Future plans	CDC	Describes planning datasets related to future uses

2.

Prompt Analysis.

The next step in the validation process involved further analysis of the prompts that elicited the most ‘similar’ results to the original. The subject of a prompt is a critical element [87], as it defines the focal point or the “what” of the prompt [79]. In architectural prompts, Ref. [53] indicates that the subject usually includes style and material layers, pertaining to the aesthetic of the building and construction materials. The second element of a prompt is the form defining the “how” [79]. This aligns with the form and environmental layers in architectural prompts, detailing the building’s shape, structural aspects, and location [53]. Including a location in a prompt can significantly influence the output of the generated images [78]. The third element, content, focuses on the “why” behind the visual representation, highlighting the intention, purpose, or meaning expressed through the design [79].

The ‘successful’ prompts from the previous stage were broken down into three main categories, i.e., subject, form, and content, as can be seen in the example in Table 2. These categories were then evaluated to understand their impact and importance. This evaluation aimed to refine the prompt template for a more precise replication of heritage buildings.

3.

Development of prompt template.

Based on the evaluation presented in Table 2, the results obtained from the development of successful/similar implementation of AI and prevailing thoughts on template generation, a standardised heritage prompt template (SHePT) was developed, as shown below. Within the template, the areas in [ ] provide the user input specific to the building.

A [number of storeys]-storey [type of building], styled in the [style/age of the building]. The building is constructed of [material/colour of the building]. Architectural elements include [specific architectural elements] made of [material of elements]. Situated in [context/environment], the building was designed by [architect]. It underwent restoration [details about restoration made] and shows signs of [any deterioration]. Currently, it is used for [current use/occupancy]. Plans for the future include [future plans]

At this stage, the prompt still contained specific areas in which architectural-focused terminology was implemented. A key factor in the creation of oral histories, however, is the omission of specific architectural vernacular when buildings are described by those who do not possess specific architectural knowledge. In order to overcome this problem, an Architectural Heritage Transformer was developed as a tool to convert plain language narratives into an architecturally specific description.

5. Creation of the Architectural Heritage Transformer (AHT) (Phase 2)

With the creation of a standardised approach to producing an AI prompt, the focus of the next phase of the framework process sought to create an ontology-based tool that will transform plain language descriptions from oral histories into architectural terminologies. This will ensure that the AI prompt is correct with respect to the architectural vernacular. The key objective is to ensure that descriptions and information gleaned from oral histories can be converted using a generated ontology. The components of an ontology, as defined by [88], are represented in the following sequence:

O = <C, H, R, A>

where O represents ontology, C represents a set of classes (concepts), H represents a set of hierarchical links between the concepts (taxonomic relations), R represents the set of conceptual links (non-taxonomic relations), and A represents the set of rules and axioms. To provide constraints to the initial establishment of the taxonomy, the development was confined to Victorian-era architecture; however, the model was developed such that it can be extensible for other eras and styles as required.

The methodology for constructing the ontology for this study is based on the NeOn methodology [89]. This has been followed by several researchers in various fields [90,91] and includes several scenarios or paths depending on the availability of the existing ontologies similar to the one being developed. Given the rich set of ontologies relevant to this research domain [92,93,94,95,96,97,98], this study will follow NeON Scenario 6, which guides the development of an ontology by reusing, merging, and re-engineering ontological resources [99]. This scenario progresses through the phases illustrated in Figure 3.

Figure 3. Ontology development phases.

5.1. Ontology Search

Initially, an extensive search for potential ontological resources that meet the specified requirements was undertaken. This search was carried out in various repositories and registries, including [100,101,102], and a diverse set of relevant ontological resources was considered for the development process. The search criteria were tailored with the aim of transforming plain language descriptions from oral histories into architectural terminologies for use in the SHePT. Out of the ontologies considered, the following were selected for the next step:

BOT (Building Topology Ontology): Focuses on the core topological concepts of buildings, including storeys, spaces, and building elements [92].

BHP (Built Heritage Properties): Provides detailed properties related to built heritage aspects, such as architectural style, heritage value, historical periods, and functions [93].

CDC (Construction Dataset Context): Describes the context in which construction data are collected and used [94].

ConTax (Construction Taxonomy): A taxonomy for organising construction-related terms and contexts [95].

MWV-D (Monumentenwacht Vlaanderen Damage Ontology): Focuses on observable damages in buildings [96].

DOT (Damage Topology Ontology): Describes the topology of damages in construction [97].

CTO (Construction Tasks Ontology): Provides a detailed ontology for construction and restoration tasks [98].

5.2. Ontology Reuse and Integration

Following the ontology search, the identified ontological resources were evaluated for alignment with this study’s requirements. To ensure these resources comprehensively address the needs of the Architectural Heritage Transformer (AHT) ontology, the following activities were performed:

(a)

Ontology Aligning.

An alignment of the selected ontological resources was performed to identify overlaps and complementary areas. This step involved mapping the concepts and properties of each ontology to ensure consistency and integration. Table 3 outlines the available user input slots within the SHePT and their corresponding relevant ontologies:

(b)

Ontology Merging.

Using the alignments identified in Table 3, the selected ontological resources were merged to create a comprehensive and cohesive AHT ontology. This process involved integrating overlapping concepts and classes and ensuring all relevant properties were included.

5.3. Requirement Specification

The aim of this step, according to [89], is to produce the Ontology Requirements Specification Document (ORSD), outlining the purpose, scope, and implementation language, as well as identifying the target audience, intended uses, and a set of requirements the ontology must meet. This is expressed primarily in the form of competency questions (CQs) [103]. The AHT ontology ORSD is displayed in Table 4.

Table 4. AHT ontology ORSD.

Component	Description
Purpose	The purpose is to convert plain language descriptions from oral histories into architectural terminologies specific to heritage buildings in England.
Scope	The AHT Ontology identifies entities representing building attributes, architectural elements, historical information, and possible current and future uses of historical buildings. It provides values for subcategories such as building type, construction materials, specific architectural elements, architect, restoration details, signs of deterioration, and current and future uses.
Implementation language	The AHT ontology is implemented in OWL/RDF using Protégé.
Intended End-Users	The AHT ontology is vital for (i) heritage professionals developing 3D models of lost heritage buildings, (ii) researchers documenting and analysing built heritage, and (iii) AI developers using heritage data to generate textual descriptions and visual representations.
Intended Uses	The AHT ontology models the architectural terminology and relationships needed to describe heritage buildings.
Non-Functional requirements	The ontology must be modular (flexible) and easily extendable to include other architectural styles and elements.
Functional requirements	The ontology should be able to answer a set of competency questions (CQs) to validate its coverage and functionality.
CQs	CQ1: General Information about a Building CQ1.1: How many storeys does the building have? CQ1.2: What is the type of the building (e.g., residential, commercial)? CQ1.3: What is the architectural style or age of the building? CQ1.4: What materials and colours are used in the building’s construction? CQ2: Architectural Elements CQ2.1: What are the specific architectural elements of the building? CQ2.2: What materials are used for the architectural elements? CQ3: Historical Information CQ3.1: Who is the architect of the building? CQ3.2: What restoration details are available for the building? CQ3.3: Are there any signs of deterioration in the building? CQ4: Current and Future Use CQ4.1: What is the current use or occupancy of the building? CQ4.2: What are the future plans for the building? CQ4.3: What historical changes in usage has the building undergone? CQ5: Context and Environment CQ5.1: Where is the building located? CQ5.2: What is the environmental context or surrounding area of the building?

5.4. Conceptualisation Phase

The conceptualisation phase involves organising and structuring knowledge into meaningful models at the knowledge level. Based on the AHT ontology ORSD, the conceptual model is structured into core concepts with detailed subclasses. Figure 4 illustrates the conceptual model, highlighting the main classes and their interrelations. Each class in the conceptual model is extracted from either the subject, form, or content of the SHePT, as described in the previous section. For example, the Building class is part of the subject of the prompt and includes core properties like the storeys, type of building, style and age of building, construction material, and colour.

Figure 4. AHT ontology conceptual model.

5.5. Formalisation Phase

This activity involves transforming the conceptual model into a semi-computable model [89]. This includes identifying concepts, the hierarchical relationships between concepts (subsumption relations), instances of concepts, and the properties or relations associated with those concepts [104]. By creating individuals, the AHT ontology can model any instance related to a heritage building. Instances or individuals, as defined by [105], are crucial for representing the most specific concepts within a knowledge base. For example, within the Architectural Elements class, instances can be added that can be found in Victorian heritage buildings such as Bay Windows, Decorative Cornices, Iron Railings, and others. To aid in accurately identifying these architectural elements, a range of sources was consulted, including the HAR 2023 register [82] and other relevant resources [106,107,108,109,110]. This process involved undertaking an analysis to identify mentions of Victorian heritage buildings and noting relevant information about architectural elements, historical details, and current and future use for inclusion in the ontology. As noted previously, in developing the AHT, the focus was directed to Victorian architecture to maintain a manageable scope, though the ontology can be expanded to include any other architectural style.

The classes and subclasses were formalised for the AHT ontology, and detailed class hierarchies were specified, ensuring all necessary subclasses were included (Table 5).

Table 5. Class hierarchies in the AHT ontology.

Key Concepts and Terms	Subcategories	Data Attributes	Description
Building Attributes	Number of Storeys	hasNumberOfStoreys: numeric	Number of floors in the building
	Type of Building	hasBuildingType: categorical	Describes the type of building (e.g., residential, commercial, industrial)
	Style of the Building	hasStyle: categorical	Describes the architectural style of the building (e.g., Victorian, Gothic)
	Age of the Building	hasConstructionYear: numeric	Records the year the building was constructed
	Construction Material	hasConstructionMaterial: categorical	Type and colour of materials used in the building
	Material Colour	hasmaterialColour: text	Colour of materials used in the building
Architectural Elements	Specific Architectural Elements	hasArchitecturalElement: text	Name of the architectural element (e.g., Bay Window, Decorative Cornices)
	Material of Elements	hasElementMaterialType: categorical hasElementMaterialColour: text	Type and colour of materials used in the architectural element
Context/Environment	Location/Environment	locatedIn: text	Description of the building’s location and surrounding environment
Historical Information	Architect	designedBy: text	Name of architect
	Restoration Details	underwentRestoration: text restorationDate: date	Details about any restoration work, including the date and description
	Signs of Deterioration	showsSignsOf: categorical	Types of any deterioration observed in the building
Current and Future Use	Current Use/Occupancy	hasCurrentUse: categorical	Details about current use of the building
	Future Plans	hasFuturePlan: text	Description of any future plans for the building

Additionally, object properties and data properties were identified. While object properties build relations between classes, data properties specify attributes of classes. For example, to detail the connections between different aspects of the building, the object property hasCurrentUse relates the Building class with the Current Use/Occupancy class, indicating the current use of the building, while the data property hasStyle associates the Building class with a string representing the architectural style.

To ensure logical consistency and computability, the constraints and relations are formalised using axioms and rules. This includes specifying domains and ranges for object and data properties and ensuring coherence within the ontology. For example, the hasArchitecturalElement property has building as its domain and specific architectural elements as its range. Table 6 presents a list of object and data properties.

Table 6. Object and data properties in the AHT ontology.

Object Property	Domain	Range
hasFuturePlan	Building	Future Plans
hasCurrentUse	Building	Current Use/Occupancy
designedBy	Building	Architect
hasArchitecturalElement	Building	Specific Architectural Element
showsSignsOf	Building	Signs of Deterioration
underwentRestoration	Building	Restoration Details
Data Property	Domain	Range
hasStyle	Building	xsd:string
hasConstructionMaterial	Building	xsd:string
hasConstructionYear	Building	xsd:int
hasElementMaterialColour	Materials of Elements	xsd:string
hasNumberOfStoreys	Building	xsd:int
hasRestorationDetails	Building	xsd:string
hasSignsOfDeterioration	Building	xsd:string
hasType	Building	xsd:string
locatedIn	Building	xsd:string

5.6. Implementation Phase

As specified in the ORSD (Table 4), the AHT ontology is implemented in OWL/RDF using Protégé. Protégé is an open-source ontology editor developed in Java and is freely available for public use and modification. It supports a variety of plugins and serves primarily to facilitate the creation and organisation of ontologies [111]. The version used for this study is Protégé 5.6.1. The implementation phase involved transforming the formalised conceptual model into a fully computable ontology that can be used for automated reasoning and data integration. This step included defining classes, properties, individuals, and relations within the Protégé environment. The AHT ontology consisted of 10 main classes, 42 subclasses, 6 object properties, and 8 data properties. The classes and subclasses included all the conceptual classes, as shown in Figure 4. After forming the classes and their respective subclasses that comprise the AHT ontology, the next step involved defining the object properties and data properties, as shown in Figure 5.

Figure 5. Class and subclass. Source: Protégé software, version 5.6.1.

Each object property connects related classes and has specified domains and ranges (Figure 6a). Data properties are used to describe the literal values associated with these classes. Figure 6b provides an illustration of the various object properties linked to the Building class. For instance, the Building class is connected to the Architect class via the designedBy object property, to the ArchitecturalElement class via the hasArchitecturalElement object property, to the Deterioration class via the showsSignsOf object property, and to the Restoration class via the underwentRestoration object property.

Figure 6. (a) Object properties and data properties implementation. (b) Building class and its relations.

6. Implementing the AI Text-to-Image Generation Toolkit (Phase 3)

In this final stage, the AHT ontology is utilised to automate the generation of a standardised heritage prompt by populating the template slots with information from the ontology. The process uses a Python script that employs the OWLready2 library to load the ontology and generate a prompt. OWLready2 is a Python module for ontology-oriented programming that serves as a gateway between Python and the Semantic Web. It allows for loading, modifying, and saving OWL ontologies, enabling the code to manipulate OWL ontologies smoothly [112]. While several tools exist for editing, aligning, or evaluating ontologies, few solutions provide a user-friendly programming interface for assessing and modifying ontologies within a programming language [113]. Traditional approaches using query languages such as SPARQL and APIs (Application Programming Interfaces) are often viewed as not user-friendly and often focus more on performance than ease of use. OWLready2, on the other hand, combines the principles of object-oriented programming with ontology management, providing an intuitive and efficient way to manipulate ontology components using Python [113]. This approach allows ontology classes to be treated as Python classes and instances as Python objects [113]. Using OWLready2, attributes like construction materials, architectural elements, styles, and architects can be extracted from the ontology, ensuring the appropriate architectural terminology is reflected in the generated prompt.

The developed script queries the ontology to fill in the template fields. OWLready2 provides various methods to query and manipulate the ontology. This library’s capabilities enable the extraction of individual instances and associated data property values from any ontology, a process seamlessly executed by the script [112].

7. Pilot Case Study Implementation

To evaluate the effectiveness of the EH-DT framework, the SHePT and the AHT, an initial implementation was conducted using the Church of St Michael, Alberbury with Cardeston, as a pilot case study. The objective of this evaluation was to assess the framework’s applicability in generating geometric twin data from intangible sources, such as oral histories. The Church of St Michael, located in rural Shropshire, England, dates back to the 12th century, with significant rebuilds in 1749 and 1844. The building, which features Gothic architecture and is constructed from uncoursed Alberbury breccia with sandstone ashlar dressings, underwent restoration in 1905 under the direction of AE Lloyd Oswell, addressing issues with the slate roof, parapet gutter, and weathervane.

As shown in the process diagram (Figure 7), the evaluation began with the collection of oral histories from individuals who possess knowledge of the building’s history and architecture. These oral histories formed the foundational dataset, captured as plain language responses (PLRs). When available, additional written or documented material about the building was incorporated to enhance, augment, and refine these responses, ensuring a more comprehensive dataset. The framework was designed to prioritise oral histories as the primary data source but remains flexible to integrate archival information wherever possible. This iterative approach allows the process to evolve, generating updated prompts and AI-generated images that progressively align more closely with the original building. This combined information was then structured and input into the EH-DT framework starting with the AHT ontology.

Figure 7. Case study implementation process.

The responses for the case study were gathered from three individuals: two professionals—a historic building surveyor and an architect—along with a member of the public with no professional architectural knowledge. As part of the process, the individuals were shown images of the building and then subsequently asked to describe its architectural and historical features based on their knowledge. The interview process was guided by the previously designed CQs. These questions, as suggested by [75], were narrow in scope to ensure that the responses provided comprehensive and focused insights into the architectural, historical, and contextual aspects of the heritage site. Alongside these descriptions, additional online resources were consulted to ensure a thorough understanding of the building’s characteristics, which were subsequently used to inform the PLRs, as shown in Table 7.

Table 7. Interview questions and plain language responses (PLRs).

Interview Questions

General Information about a Building CQ 1.1—How many storeys does the building have? The church had one storey. CQ 1.2—What is the type of the building (e.g., residential, commercial)? It was a church. CQ 1.3—What is the architectural style or age of the building? The architectural style was a mix of 12th-century origins with significant rebuilds in 1749 and 1844. It features elements of Gothic architecture. CQ 1.4—What materials and colours are used in the building’s construction? The church was constructed with uncoursed Alberbury breccia with sandstone ashlar dressings. The roofs were mostly covered by plain clay tiles, and the tower roof was covered with slate. Architectural Elements CQ 2.1—What are the specific architectural elements of the building? The church had a nave and chancel built in 1749, a semi-detached west tower added in 1844, Gothic windows, and a decorative weathervane. CQ 2.2—What materials are used for the architectural elements? The nave and chancel were made of uncoursed Alberbury breccia with sandstone ashlar dressings. The Gothic windows were framed in sandstone, and the weathervane was metal. Historical Information CQ 3.1—Who is the architect of the building? The original C12 church’s rebuild in 1749 and the addition of the west tower in 1844 do not have recorded architects, but the restoration in 1905 was by AE Lloyd Oswell. CQ 3.2—What restoration details are available for the building? The church underwent significant restoration in 1905 by AE Lloyd Oswell. Restoration included repairing the slate roof of the tower, renewing the parapet gutter, overhauling the decorative weathervane, and improving access to the tower parapet gutter. CQ 3.3—Were there any signs of deterioration in the building? Yes, the slate covering on the tower roof had failed and needed to be re-laid. The parapet gutter required complete renewal, and the decorative weathervane needed overhauling. Current and Future Use CQ 4.1—What is the current use or occupancy of the building? The church is still used for religious services and community events. CQ 4.2—What were the future plans for the building? Fundraising is in progress to address the needed repairs, including re-laying the slate roof, renewing the parapet gutter, overhauling the weathervane, and improving access to the tower parapet gutter. CQ 4.3—What historical changes in usage has the building undergone? None Context and Environment CQ5.1—Where is the building located? The church is located in Alberbury with Cardeston, Shropshire, England. CQ5.2—What is the environmental context or surrounding area of the building? The church is surrounded by rural countryside.

Once the PLRs were gathered and refined, a new instance for the Church of St Michael was created in the AHT ontology using the Protégé platform. This instance was associated with predefined object properties and data properties, such as ‘hasNumberofStoreys’ (set to 1), ‘designedBy’ (set to AE Lloyd Oswell), and ‘hasConstructionMaterial’ (set to Alberbury_Breccia). The AHT ontology is designed to include explanations and definitions for specific architectural terms, enabling users to reference these descriptions when necessary. This feature ensures consistency in the input process by offering standardised terminology and explanations. For example, users inputting details about the building materials such as the use of ‘uncoursed Alberbury breccia’ can select the most appropriate terms from the ontology based on the combined descriptions provided by previous occupants and historical documentation, making use of the definitions provided in the ontology (Figure 8).

Figure 8. Implementation of the ontology using Protégé for the pilot study.

The structured data from the AHT enabled the automatic generation of a SHePT prompt using Python code, as can be seen in Figure 9.

Figure 9. Generated heritage prompt using the AHT ontology and OWLready2.

The SHePT was then used to generate AI diffusion models through various AI text-to-image tools (Table 8). These models were subjected to a visual comparative analysis against original photographs from sources such as Historic England to assess the accuracy of the architectural details, materials, and historical features depicted by AI.

Table 8. SHePT outcomes.

Church of St Michael, Alberbury with Cardeston—Shropshire (UA)
Original Photographs from Historic England	Source: Historic England [21]
SHePT input	A [church], styled in the [c12]. The building is constructed of [of uncoursed Alberbury breccia with sandstone ashlar dressings and roofs mostly covered by plain clay tiles]. Architectural elements include [nave and chancel]. Situated in [vibrant green landscape]. It underwent restoration [in 1905 by AE Lloyd Oswell].
AI Text-to-Image Tool	SHePT-based Output
DALL·E 2
Stable Diffusion
Adobe Firefly

The validation process is iterative; after the initial review and comparison of the AI-generated models, revisions may be made to improve accuracy, as explained in Figure 7. The models are then validated by presenting them to individuals familiar with the building, ensuring that the AI-generated representations align with their recollections. This cyclical review and validation process allows for the continuous refinement of the visual element of the geometric digital twin, ensuring that the EH-DT framework remains a robust tool for the accurate digital reconstruction of lost or damaged heritage buildings. The results, validated against historical records and expert feedback, demonstrated the framework’s capacity to produce reliable digital representations that contribute to heritage conservation efforts.

8. Discussion

The development and application of the EH-DT framework mark a significant advancement in the field of architectural heritage, particularly in addressing the challenges associated with lost or damaged heritage buildings. This study introduces a structured approach to incorporate intangible data sources, such as oral histories, into the digital reconstruction process. By leveraging AI text-to-image generation tools, the framework facilitates the recreation of architectural elements for heritage sites where physical records are incomplete or non-existent.

One of the key findings of this study is the effectiveness of the SHePT in generating accurate architectural representations from intangible sources. While the AHT ontology was developed as part of the overall EH-DT framework, it also holds the potential to be used independently to convert plain language descriptions into structured architectural terminology. Additionally, the ontology can be expanded to include more comprehensive heritage information, covering a broader range of architectural styles and historical periods, making it an extensible and versatile tool for various heritage conservation efforts. The pilot case study demonstrated the potential of the framework to generate visual reconstructions closely aligned with original building characteristics, as validated through community feedback and archival comparisons.

However, its reliance on generative AI introduces challenges, particularly the potential for speculative or inaccurate outputs. Those tools might extrapolate or ‘invent’ details, which may lead to inaccurate representations of historical buildings. While the iterative validation process mitigates this risk to some extent, it cannot fully eliminate uncertainties. Therefore, AI-generated models should be regarded as supplementary tools requiring validation through community engagement and archival corroboration.

This study also reveals notable challenges. The reliance on oral histories introduces subjectivity, as recollections may vary and lack precise architectural details. This highlights the necessity of iterative feedback loops with community members and experts to refine AI-generated outputs, mitigating discrepancies through continuous validation against archival materials and expert reviews. Additionally, while the current framework generates 2D representations, the next stage of development will focus on extending these outputs into 3D environments, forming the foundation for future geometric digital twins. This evolution will require integrating operational and environmental data, paving the way for Echo-based Operational Digital Twins that reflect the functional dynamics of historic buildings.

Additionally, while the framework shows promise, its scalability remains an area for further investigation. The case study evaluation focused on a single building with a relatively small set of oral history respondents. Applying this framework to a wider range of buildings, including larger, more complex heritage sites, will require the integration of more diverse data sources to ensure the fidelity of the digital reconstructions. A key area for future development is applying the framework to hybrid cases, where some elements of the lost heritage building remain intact. By combining physical data from existing structures with historical operational data and insights from previous occupants, the EH-DT framework could progress towards higher levels of digital twin maturity. In such cases, sensor technologies could be deployed within surviving structures to complement intangible data, enabling a more comprehensive and dynamic representation of heritage assets.

9. Conclusions

The aim of this study was to develop a novel approach that extends beyond the existing tangible-based HBIM methods towards an ‘Echo-based’ Heritage Digital Twin (EH-DT) generated using AI techniques. By utilising intangible data sources such as oral histories, the framework transforms these narratives into structured architectural data for digital reconstructions of lost or partially damaged heritage buildings.

The integration of AI-generated imagery with structured ontological data represents an advancement in preserving both the tangible and intangible characteristics of heritage buildings. Although the current framework focuses on producing 2D representations, it lays the foundation for future advancements into 3D modelling and comprehensive digital twin systems, contributing to the evolution of HBIM into more dynamic heritage management tools.

Nonetheless, this approach has limitations that must be acknowledged. Generative AI tools, while innovative, may extrapolate details that compromise historical accuracy. Despite iterative validation processes, uncertainties cannot be fully eliminated. Therefore, AI-generated outputs should be regarded as supplementary tools rather than definitive reconstructions.

By integrating intangible cultural heritage into digital workflows, the EH-DT framework offers a scalable tool for heritage conservation projects. This approach supports the preservation and understanding of cultural heritage through digital innovation, bridging the gap between memory-based narratives and formal architectural reconstructions.

While the framework shows potential, future research will focus on expanding the scale and scope of its application to more complex heritage partially lost heritage sites. Advancing towards fully realised digital twins may involve incorporating operational and environmental data, supported by sensor techniques. This iterative, community-driven process ensures that the EH-DT framework not only supports heritage preservation but also encourages engagement with local communities, ensuring that collective memories and lived experiences contribute to safeguarding cultural heritage for future generations.