Metaverse and Digital Twins in the Age of AI and Extended Reality

Abstract

This paper explores the evolving relationship between Digital Twins (DT) and the Metaverse, two foundational yet often conflated digital paradigms in digital architecture. While DTs function as mirrored models of real-world systems—integrating IoT, BIM, and real-time analytics to support decision-making—Metaverses are typically fictional, immersive, multi-user environments shaped by social, cultural, and speculative narratives. Through several research projects, the team investigate the divergence between DTs and Metaverses through the lens of their purpose, data structure, immersion, and interactivity, while highlighting areas of convergence driven by emerging technologies in Artificial Intelligence (AI) and Extended Reality (XR).This study aims to investigate the convergence of DTs and the Metaverse in digital architecture, examining how emerging technologies—such as AI, XR, and Large Language Models (LLMs)—are blurring their traditional boundaries. By analyzing their divergent purposes, data structures, and interactivity modes, as well as hybrid applications (e.g., data-integrated virtual environments and AI-driven collaboration), this study seeks to define the opportunities and challenges of this integration for architectural design, decision-making, and immersive user experiences. Our research spans multiple projects utilizing XR and AI to develop DT and the Metaverse. The team assess the capabilities of AI in DT environments, such as reality capture and smart building management. Concurrently, the team evaluates metaverse platforms for online collaboration and architectural education, focusing on features facilitating multi-user engagement. The paper presents evaluations of various virtual environment development pipelines, comparing traditional BIM+IoT workflows with novel approaches such as Gaussian Splatting and generative AI for content creation. The team further explores the integration of Large Language Models (LLMs) in both domains, such as virtual agents or LLM-powered Non-Player-Controlled Characters (NPC), enabling autonomous interaction and enhancing user engagement within spatial environments. Finally, the paper argues that DTs and Metaverse’s once-distinct boundaries are becoming increasingly porous. Hybrid digital spaces—such as virtual buildings with data-integrated twins and immersive, social metaverses—demonstrate this convergence. As digital environments mature, architects are uniquely positioned to shape these dual-purpose ecosystems, leveraging AI, XR, and spatial computing to fuse data-driven models with immersive and user-centered experiences.

1. Introduction: Digital Twins (DTs) and Metaverse

Digital Twins (DTs) and the Metaverse have emerged as rapidly evolving digital paradigms within architectural practice and education in recent years. While both involve creating and interacting within virtual environments, they operate within distinct conceptual and functional frameworks. Understanding their differences is essential for leveraging their design, analysis, and human engagement. The research questions are as follows: How are DT and the Metaverse converging through advancements in Artificial Intelligence (AI), Extended Reality (XR), and spatial computing? And what implications does this hybrid integration have for digital architecture, user engagement, and architecture education? Thus, the goal of this paper is to investigate the overlapping but distinct roles of Digital Twins and the Metaverse in architecture, especially in the context of AI and XR. Through comparisons of conventional and AI-driven workflows and presenting real-world applications, the team is able to examine how these two paradigms both converge and differ in different areas of architectural work.

1.1. DTs as Mirrored Reality

A DT is a virtual model that mirrors a physical system or process, continuously updated through real-time data. It enables simulation, analysis, and performance forecasting of its physical counterpart. By integrating diverse data sources, such as Internet of Things (IoT) sensors and Building Information Modeling (BIM), DTs enhance system understanding, support informed decision-making, and optimize operations throughout the lifecycle of a built asset. Van der Horn and Mahadevan (2021) define a DT as a “generalized framework that characterizes and implements virtual representations of physical entities to support decision-making through real-time data integration and predictive analytics” [1].

DT development relies on reality-capture technologies, including geometric data acquisition through point-cloud scans [2], semantic and time-series data fusion [3], and metadata management. Among these, the integration of BIM remains a dominant methodology in the AEC industry, serving as a foundational framework for accurate and dynamic digital representations [4]. Computer vision technologies further bridge the gap between BIM and DTs, enabling real-time synchronization and the creation of “living” digital models [5].

Integrating IoT technologies into DT systems redefines smart buildings’ design, operation, and adaptability. Popescu et al. (2022) emphasize the inclusion of environmental functions for sustainability [6], while Eneyew et al., (2022) highlight dynamic responsiveness enabled by IoT-BIM fusion [7]. Walczyk and Ożadowicz (2024) explore how distributed IoT networks address functional challenges in smart systems [4]. Collectively, these studies illustrate the transformative role of DTs in shaping sustainable and intelligent environments.

Another defining characteristic of DTs is their bi-directional interaction with physical systems. The National Academies of Sciences, Engineering, and Medicine (2024) describe DTs as “a set of virtual information constructs that mimics the structure, context, and behavior of a natural, engineered, or social system (or system-of-systems),… The bi-directional interaction between the virtual and the physical is central to the digital twin [8]”. This DT concept has extended into various fields, from architecture and urban planning to geoscience and healthcare, exemplified by innovations like patient-specific twins [9] and the “Virtual You [10]” model.

1.2. Metaverse as a Fictional World

In contrast, Metaverses typically refer to immersive, imagined environments designed for multi-user experiences rather than mirroring real-world systems. Cheng (2023) defines the Metaverse as a complex infrastructure, often integrating blockchain technologies to support decentralized ownership, digital assets (e.g., NFTs), and novel social and legal structures [11]. It is commonly portrayed as a “persistent, interconnected network of 3D virtual worlds [12]”, a “multi-user digital realm blending physical and virtual realities [13]”, and a shared environment where avatars navigate diverse experiences, maintain digital identities, and manage virtual assets [14].

This immersive space enables users to engage in “lifelike personal and business experiences online [15]”, closely mirroring real-life interactions. Widely considered the next stage of the internet, the metaverse frequently intersects with Web 3.0 and blockchain technologies, addressing legal, ethical, and economic challenges alongside applications like NFTs [11]. Its applicability and growing relevance spans various industries, including remote medicine [16], education, and banking [17]. Moreover, it plays a role in addressing broader societal issues, with geopolitical crises acting as “cultural catalysts for socio-ecological innovation” and reshaping civic landmarks in physical and metaverse spaces [18]. Park (2022) categorizes the metaverse into its key components—hardware, software, and content—and its approaches, including user interaction, implementation, and application [19]. Today, popular metaverse platforms include Meta Horizon, Epic’s Fortnite, Second Life, Roblox, Mozilla Hubs, Microsoft Mesh, Glue, and more.

2. Divergence of Metaverse and Digital Twin Approach

Although both DTs and the Metaverse inhabit virtual environments, their purposes and implementations differ significantly.

2.1. DT Applications

DTs are primarily used to replicate high-fidelity real-world systems, especially in the architecture, engineering, construction, and operation (AECO) industries. Their most prominent applications lie in post-construction building management, where they optimize energy consumption, support predictive maintenance, and enable sustainable operations. Bjørnskov and Jradi (2023) propose that a DT is an ontological energy modeling framework that enhances the scalability and adaptability of energy management [20]. Similarly, Cespedes-Cubides and Jradi (2024) illustrate how DTs improve energy efficiency by integrating real-time data with predictive analytics [21]. Li and Wang (2024) explore cost-effective DT deployments for managing existing buildings, democratizing energy optimization across diverse facilities [22]. Bortolini et al. (2022) comprehensively reviewed DT applications in terms of energy efficiency, emphasizing their role in reducing energy consumption and operational costs [23]. Beyond operational efficiency, DTs are pivotal in driving sustainable design by embedding environmental considerations into the building lifecycle. Additionally, Popescu et al. (2022) advocate for enhancing DT frameworks to incorporate functions such as resource optimization and waste reduction, underscoring the role of DTs in fostering data-driven decision-making and environmentally conscious operations [6]. DTs are also transforming the preservation of historic buildings and digital heritage, with successful cases such as the City Theatre of Norrköping [24] and the VERBUM framework for heritage preservation [25]. These diverse applications highlight DTs’ versatility and transformative potential in the AECO industry.

DTs are also central to smart city management and planning at the urban scale, integrating Geographic Information Systems (GISs), IoT, and autonomous systems. Barzilay (2018) highlights the pivotal role of DTs in advancing infrastructure for autonomous vehicles by enabling real-time data integration and management in smart cities [26]. Similarly, Tao (2013) emphasizes the significance of interdisciplinary Urban GISs for smart cities, which facilitate spatial data analysis to optimize urban operations and inform decision-making processes [27]. The MIT Media Lab’s City Science Group and Larson (2018) underscore the transformative potential of augmented reality and data convergence in creating more interactive and adaptive urban environments tailored to citizens’ needs [28,29]. Building on these ideas, Miller et al. (2021) explore the integration of DTs with GIS, BIM, and IoT systems, proposing the concept of the Internet-of-Buildings (IoB) to enhance urban-scale data integration, including wearable and IoT technologies [30].

DT applications also span diverse domains such as urban mobility, infrastructure management, and public engagement, providing innovative solutions to complex urban challenges. Furthermore, DTs are increasingly leveraged as AI training engines to generate synthetic datasets, enabling AI models to learn spatial navigation through trial-and-error methods already applied in Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs).

2.2. Metaverse Applications

The Metaverse, in contrast, emphasizes simulated social, cultural, recreational, and economic experiences, often unconstrained by physical-world replication. Specifically, TheMetaverse offers architects a “novel, unconstrained digital space” where architects can reimagine the space without physical limitations. There is an example in the Liberland Metaverse designed by Zaha Hadid Architects, a “cyber-urban” metaverse [31], where “residents can purchase plots and interact as avatars, featuring hyper-realistic districts that promote urban self-governance and spontaneous order” [32]. Patrick Schumacher used it as a demonstration to emphasize the role of architects in “designing immersive, communicative spaces and proposing frameworks for integrating virtual and physical environments” [33].

In the AECO industry, metaverse applications are also increasingly aligned with the “building lifecycle” [34], offering platforms for real-time collaboration, design visualization, and digital project management. Additionally, virtual real estate is emerging as a key focus, blending “decentralized ownership” models with blockchain-based economic ecosystems [35].

The Metaverse also offers a promising emerging platform for online education. Our research team examined its potential to augment architectural pedagogy—the educational philosophy and methods behind how architecture is taught—through immersive and interactive learning. Several platforms—including Microsoft Mesh, Virbela, Hyperspace, and Mozilla Hubs—were evaluated based on their level of customization, communication tools, 3D asset sharing capability, support for large-scale events, video conferencing support, and cross-device access. The team also considered scalability, server performance, and the integration of AI technologies, including Large Language Models (LLMs) and support for non-player-controlled characters (NPCs). The Metaverse also offers a compelling platform for online education. Our research team examined the potential of the Metaverse to augment architectural pedagogy through immersive and interactive learning. Several platforms—including Microsoft Mesh, Virbela, Hyperspace, and Mozilla Hubs—were evaluated based on their level of customization, communication tools, 3D asset sharing capability, support for large-scale events, video conferencing support, and cross-device access.

Fifteen fourth-year undergraduate architecture students participated in weekly sessions using these platforms, experiencing a game-like, collaborative environment to explore design ideas. The studio’s focus on virtual architecture prompted students to explore spatial experiences from a non-traditional perspective that is beyond typical physical constraints. In-Metaverse discussions, critiques, and walkthroughs of worlds designed by the students enabled them to evaluate how this approach differs from traditional architecture. (Figure 1)

Students demonstrated increased fluency in metaverse-based workflows using the Metaverse for studio activities, Generative AI, and deploying their design in their own Metaverses. Structural elements such as columns or beams were reinterpreted for symbolism or familiarity, while features like stairs and ramps were employed to enhance navigation and spatial understanding.

Although challenges remained, such as accessibility barriers, technical glitches, and uneven user experiences, the platforms showed strong potential for immersive education and real-time collaboration. Ultimately, the Metaverse offers an emerging platform for speculative architectural exploration and digital co-creation, expanding how future architects engage with design and virtual environments.

3. Interaction of AI with DT and Metaverse

Although DT and the Metaverse have distinct purposes, both continue to be transformed by advancements in AI and Extended Reality (XR). Historically rooted in earlier digital and gaming worlds, the metaverse has expanded significantly with advancements in AI and XR technologies, as Esen et al. (2023) note [36]. By merging physical and virtual worlds through immersive technologies like virtual reality (VR), augmented reality (AR), and mixed reality (MR), the metaverse has the potential to “revolutionize social, technological, and behavioral domains” while addressing challenges such as accessibility, security, and user immersion [37].

This section explores how AI transforms DT creation and Metaverse development, particularly machine learning, generative AI, and Large Language Models (LLMs). (Figure 2). The team also examined their application in environment simulation, management, and user interaction.

3.1. AI for Content Creation

3.1.1. Traditional vs. AI-Driven DT Workflows

Conventional DT creation typically begins with capturing a physical environment using structured, high-precision data sources such as LiDAR scanning, photogrammetry, and BIM. In our case study of the UC Digital Futures building, these approaches enabled the construction of a detailed digital replica of the building. However, challenges emerged around data interoperability across various BIM platforms and DT systems. The team evaluated three key platforms—Autodesk Tandem, NVIDIA Omniverse, and Unreal Engine—all offering varying degrees of BIM integration. Tandem provided the most seamless solution, offering cloud-based BIM synchronization that maintained metadata, system hierarchies, and granular building components like HVAC, electrical, and plumbing systems. Omniverse enabled high-quality visualization using USD format and RTX acceleration, while Unreal Engine excelled at creating immersive, metadata-rich environments via its Datasmith pipeline (

Table 1. Evaluation of various Digital Twin (DT) platforms. The highest performing platform in each category is highlighted in green.

	Autodesk Tandem	NVIDIA Omniverse	Unreal Engine
BIM integration	Native, with Metedata	Convert to USD	Datasmith
IoT integration	Native with AWS, Azure	Python	Python
Bi-directional data exchange	Customized API	Customized API	Customized API
XR Support	Limited	VR	VR, AR
LLM integration	Limited, through widgets	NVIDIA ACE	Plugins
Visualization	Web3D rendering	High-fidelity	High-fidelity

).

The team incorporated IoT sensor data streams to enhance functionality and monitor parameters such as temperature, humidity, air quality, and lighting. Tandem’s built-in support for AWS and Azure cloud services made it especially effective for real-time building performance tracking. Omniverse and Unreal required additional customization and scripting to support similar real-time analytics and dashboard development. While robust, the traditional DT workflow remains resource-intensive, involving multiple data pipelines, hardware setups, and bespoke platform configurations to achieve real-time feedback and visualization. As illustrated in Figure 3, the team investigated multiple DT methods, including LiDAR, BIM-based workflows, and the emerging GS technique—all integrated into a unified digital twin framework overlaid with IoT data.

An emerging alternative to the reality capture pipeline is Gaussian Splatting (GS), a machine-learning-based technique for 3D reconstruction. The team tested GS on a smaller-scale architectural installation—a Sukkah pavilion. Unlike traditional modeling methods that rely on geometry and mesh construction, GS represents a scene as a collection of 3D Gaussians, encoding spatial and visual information such as position, scale, orientation, color, and transparency. This volumetric rendering approach allows for continuous, photorealistic rendering with impressive accuracy and efficiency, particularly in dynamic lighting conditions.

Machine learning drives the optimization process in GS. Using differentiable rendering and neural networks, the algorithm minimizes the difference between rendered views and actual photographs captured from multiple angles. The team generated a high-resolution DT model using video and photo sequences in a fraction of the time required by conventional methods, with minimal manual intervention.

This process eliminated the need for LiDAR or BIM inputs, making it ideal for rapid prototyping and informal structure visualization. As GS matures, its potential for intelligent, AI-enhanced environmental simulations is significant, offering new opportunities for real-time, context-aware DTs. While traditional DT workflows are well-suited for large-scale, infrastructure-rich environments, Gaussian Splatting introduces a flexible, AI-driven alternative optimized for speed, realism, and accessibility. Together, these methods form a complementary toolkit for various DT applications, from structured asset management to experimental, lightweight visualization.

3.1.2. Metaverse Creation with Generative AI

Artificial Intelligence-Generated Content (AIGC), especially in speculative and fictional contexts, plays a central role in shaping the Metaverse by streamlining the creation of virtual assets and immersive environments. Generative AI tools enable scalable content production through prompt-based workflows and automation, transforming how virtual environments are conceived and built.

AI has been applied in the AEC industry through “generative design, GANs, evolutionary algorithms (GAs), and integrating BIM databases with GPT” [38]. Various AI tools released architects from routine execution to creative exploration and refinement. While DT uses LLM to assist humans in interacting with the vast dataset from IoT data, the Metaverse uses AI “to influence areas such as creativity, leadership, and customer satisfaction” [39].

Metaverse creation has deep roots in video games as a fictional virtual environment. Metaverse uses “video game development, and storytelling to create immersive virtual experiences, particularly in the context of enterprise applications” [40], or as Alice Bucknell described, “gaming technologies and speculative storytelling” to build a “Multispace” [41]. Tools like Stable Diffusion, Midjourney, and 3D modeling platforms like Rodin, Trellis, and Tripo have expanded access to virtual asset creation. Generative AI bridges the gap between conventional architectural tools and immersive digital workflows by converting simple text or image prompts into game-ready models. It also enables parametric modeling, allowing users to design complex forms with minimal technical overhead.

Unlike DTs, which rely on real-world data for replication and monitoring, the Metaverse is typically untethered from physical constraints. AI-generated assets—from text-to-3D models to image-to-3D—can be produced rapidly, bypassing time-intensive traditional modeling. Despite the bias and hallucination problem that AI sometimes has, the “user control and AI automation with generative AI technologies are shaping the Metaverse, transforming it into a dynamic, immersive, and interactive virtual world” [42].

Our research investigated how Gen-AI tools transform architectural visualization and digital content generation. The team tested platforms such as Stable Diffusion, Midjourney, DALL·E, Adobe Firefly, PromeAI, Vizcom, Krea, and more to evaluate their ability to produce architectural imagery. Factors affecting quality included rendering styles (2D/3D), material types (e.g., sketch, photorealism), references to architects or styles, and scene descriptors, including lighting and environmental context.

For example, a prompt like “Create a photorealistic 3D rendering of a futuristic suspension bridge in the style of Zaha Hadid, predominantly white, under dynamic lighting at dusk over a scenic river” generated highly coherent results. PromeAI and Vizcom enabled sketch-to-render workflows, while ControlNet paired with ComfyUI helped convert linework diagrams into detailed visualizations. The team also trained custom LoRA models within Stable Diffusion to replicate specific design vocabularies, offering refined style control.

The team further tested text-to-3D tools like Meshy, Tripo, and DreamGaussian using prompts like “a soft, cream-colored couch”. While visually rich, these tools often struggled with spatial logic and architectural coherence, making them better suited for schematic design and object-level prototyping. Additional 2D-to-3D platforms like Rodin, Trellis, and Hunyuan varied in performance; DreamGaussian excelled at basic forms but required longer processing, whereas Tripo delivered faster, though less geometrically accurate, results. The team also evaluated how abstract Midjourney renderings were translated into 3D and optimized across platforms for 3D printing (

Table 2. AI tools for content creation. More details are available at http://ming3d.com/new/2024/12/28/arch-viz-3-2024/ (accessed on 23 May 2025).

Category	2D Image Generation Tools	3D Gen-AI Tools	Gaussian Splatting Tools
Tools	Stable Diffusion MidJourney, DALL·E, Adobe Firefly, Hunyuan, PromeAI, Krea, Vizcom, etc	Rodin, Meshy, Tripo, Dream Gaussian, Trellis	VastGaussian, CityGaussian, OccluGaussian
Features	Text prompts, image inpainting, upscaling, and stylized output	Text-to-3D, image-to-3D, rapid prototyping to generate 3D mesh models	3D volumetric model
UI	Discord-based, web UI, ComfyUI, ControlNet, Lora	Web UI, ComfyUI	Standalone ML training software

).

Though not a Gen-AI model per se, Gaussian Splatting was included in our comparison due to its value in converting image and video input into immersive, high-fidelity 3D environments. With strong spatial accuracy and real-time rendering capabilities, it complements generative tools for Metaverse scene construction.

Despite the promise of generative AI technologies in architectural workflows, their relative immaturity and experimental nature currently limit their scope to early-stage schematic design and object-level prototyping. While tools like Midjourney, Stable Diffusion, and Tripo can quickly generate visually compelling assets, our testing revealed frequent spatial inaccuracies and a lack of architectural coherence, even at the single-object level. These issues are further observed when attempting to generate complex, multilayered architectural compositions, making it difficult to achieve outputs that meet industry standards.

As such, relying solely on these tools for large-scale or construction-ready projects remains unfeasible. However, their speed and accessibility make them valuable in supporting ideation, conceptual visualization, and preliminary design exploration. A more approachable pipeline may involve using generative tools for initial asset generation, followed by manual refinement and compilation within professional-grade modeling environments. Ultimately, their strength lies in changing the schematic and preparatory phases of architectural design, rather than replacing traditional design development and documentation processes.

In summary, continued development is essential to support high-resolution, spatially coherent, and large-scale metaverse environments, paving the way for more intelligent, immersive, and creative digital worlds.

3.2. AI for Management

3.2.1. LLMs for DT Management

LLMs are being increasingly integrated into DT systems to support performance analysis, predictive maintenance, and natural language-based data querying. These AI agents allow users to engage with complex datasets through intuitive natural-language-based interactions, making system monitoring and decision-making more accessible and intelligent. The integration of AI and LLMs is transforming DT functionality across multiple domains. Ferdousi et al., (2024) demonstrate this synergy in railway defect inspection through DefectTwin, where LLMs analyze real-time monitoring data to enhance defect detection and predictive maintenance [43]. In the context of power and environmental systems, Yang et al., (2024) propose an LLM-enhanced multi-agent system for DTs, enabling comprehensive risk analysis by combining expert knowledge with multi-dimensional data-driven insights [44]. Additionally, the incorporation of “human-in-the-loop” frameworks, as discussed by Yang, Siew, and Joe-Wong (2024), optimizes decision-making in DTs by leveraging LLMs to mediate between human expertise and complex system simulations [45]. Šturm et al., (2024) explore the use of LLM agents to enhance the cognitive capabilities of DTs for countries, enabling more intuitive and intelligent interactions to support decision-making and policy analysis at a national scale [46].

Building on these developments, our research team created an experimental framework for enhancing DT management with LLM agents. This system features multiple specialized agents capable of observing, reasoning, and interacting with DT simulations in real time. The team focused on how multi-agent LLM systems can enhance IoT data visualization and decision support, and how natural language interactions enable context-aware communication [47]. The framework supports predictive maintenance by analyzing IoT data to identify anomalies and forecast potential failures, reducing operational downtime and extending equipment life cycles. LLM-powered conversational interfaces offer expert-level guidance, real-time visualization, and diagnostic support (Figure 4).

To address the evaluation of LLMs in DT contexts more directly, performance and usability were assessed based on their functional roles within existing digital twin projects. Specifically, evaluation focused on two key criteria: (a) the accuracy of LLM agents in analyzing real-time IoT data, detecting anomalies, and generating effective failure forecasts; and (b) the breadth of their usability across various decision-making and user interaction scenarios. Multi-agent LLM systems were tested for their ability to enhance system-level reasoning and visualization, enabling more informed, context-aware decisions.

Platform comparisons, such as Unreal Engine versus Autodesk Tandem, also revealed how interface capabilities affect the practical integration of LLMs. Another important evaluation metric was their effectiveness as mediators in human-in-the-loop workflows. LLMs not only supported experts in interpreting complex simulations but also supported interaction for non-expert users by adapting to specific use cases. Together, these factors offered a comprehensive evaluation on how LLMs perform in real-time DT applications and how flexibly they can be integrated into diverse management tasks.

LLMs are embedded into DT platforms, significantly enhancing real-time analytics, fault prediction, and user interaction. Conversational AI improves usability and strengthens decision-making, with Unreal Engine demonstrating the most significant potential for immersive, LLM-enhanced DT environments.

3.2.2. LLM for Metaverse

One of the most transformative developments in the Metaverse is the integration of LLMs with independent intelligent agents, or NPCs, capable of autonomous, human-like interactions. These AI-driven NPCs introduce new interactivity, realism, and engagement levels, unlocking significant potential across gaming, education, collaborative design, and simulation-based training domains. Our investigation across a range of Metaverse platforms reveals a growing trend in embedded conversational AI, particularly models like ChatGPT-4o, into open-ended user interactions, guided by predefined behaviors.

Platforms like Hyperspace are at the forefront of this movement, offering intuitive tools for customizing conversational NPCs that integrate directly with foundational LLMs. These agents can be configured to act as educators, guides, or collaborative partners, enhancing immersion and user experience. Similarly, game engines like Unreal and Unity provide robust frameworks for deploying AI avatars, leveraging built-in systems for character animation, real-time voice synthesis, facial expressions, and environmental awareness. In parallel, companies like NVIDIA are pushing the frontier with platforms like ACE, which support speech, gesture recognition, and emotional expression in AI-enhanced digital personas.

In our lab, the team developed a prototype LLM-powered talking bot using Unreal Engine, merging spatial sensing with language-based interaction. The bots could identify user presence and location via camera input and respond using a fine-tuned GPT model for contextual awareness. This prototype demonstrated the potential of combining spatial computing and LLMs to create a responsive, semi-autonomous agent that could speak and understand its surroundings, detect human proximity, interpret room context, and react dynamically to user behavior. The system leveraged Unreal’s built-in animation, audio, and camera frameworks to deliver personalized, voice-based interactions—illustrating how conversational AI can be embedded into spatial environments to guide users and simulate natural engagement, such as in a lab tour scenario (Figure 5).

The utility of LLM-powered NPCs extends far beyond gaming. They can serve as virtual tutors in educational settings—answering questions, explaining concepts, and providing real-time feedback. In architecture and design simulations, they might function as virtual clients or stakeholders, prompting students to articulate and defend design decisions. In healthcare or emergency response training, LLM agents can take on roles like patients, instructors, or teammates, enabling realistic multi-user scenarios without requiring human actors.

Despite their promise, deploying LLMs within the Metaverse presents notable challenges. Real-time responsiveness depends on a robust infrastructure to manage token size, latency, and integration complexity. Privacy, safety, and content moderation are essential considerations, especially in open environments where user-generated input shapes AI behavior. Moreover, ethical concerns around hallucinations and perceived intelligence require thoughtful design and oversight as LLM capabilities grow.

Given these limitations, it became essential to systematically evaluate how LLMs perform in immersive, real-time environments. To evaluate the performance and usability of LLMs in Metaverse applications, multiple factors were considered. A primary metric involved the realism and engagement level of human-like interactions, assessing whether LLM-powered agents could simulate natural, autonomous, and context-aware conversations. Special attention was given to spatially situated use cases, where LLMs interacted with users based on camera input, environmental cues, and proximity detection, enabling personalized experiences, such as lab tour guides or room-aware assistants.

Another key evaluation dimension was the flexibility of LLMs across diverse domains. Applications ranged from NPCs acting as virtual tutors and design critics to roles in healthcare and emergency response training, where they simulated patients or teammates to support immersive learning scenarios.

Finally, technical and ethical considerations formed an essential part of the evaluation. This included testing real-time responsiveness under varying system loads, managing privacy concerns, and addressing risks such as hallucinations or inappropriate content generation. Systems were also assessed for their ability to implement safety guardrails and content moderation mechanisms.

Together, these evaluation criteria provided a comprehensive understanding of how LLMs perform in real-time, immersive Metaverse environments and how they can support adaptive, intelligent, and ethically sound virtual interactions.

Integrating LLMs suggests a shift toward more adaptive and personalized virtual environments. These agents are conversational partners and spatially aware collaborators offering new modes of user interaction, learning, and design support. As technology evolves, LLMs are poised to become foundational elements of next-generation Metaverse platforms, enabling rich storytelling, advanced simulation, and more human-centric digital interactions.

4. Extended Reality

As the diagram in F2 illustrates, while DT and Metaverse exist within virtual environments, they diverge significantly in terms of purpose, structure, and modes of user interaction. DTs prioritize mirroring physical systems through real-time, data-driven simulations and often do not require immersive engagement. Conversely, Metaverse emphasizes immersive, interactive experiences, where user embodiment and presence are essential, particularly in platforms like Meta Horizon or within the gaming industry like Fortnite, where sustained user engagement is critical.

Although DTs are often visualized in 3D, they traditionally rely on 2D dashboards, charts, and diagrams to represent real-time or historical IoT data. These visualizations prioritize quantitative insights over spatial or sensory immersion.

4.1. Virtual Reality for Digital Twins

DTs benefit from Extended Reality (XR) as mirrored models of physical environments by enabling immersive behavioral simulations. Our research used Virtual Reality (VR) to study human behavior in wayfinding and emergency evacuation scenarios based on a real environment. The team gained insights into spatial cognition and decision-making by capturing user locomotion data within a DT with simulated fire and smoke [48]. Utilizing a treadmill system and foot-mounted motion sensors, the team created a one-to-one spatial mapping of the real and virtual world, allowing for physically accurate walking, orientation, and speed variation (Figure 6).

The team also collected data on user navigation patterns, such as position, orientation, path, and time to reach the gathering point, to identify behavioral trends. These immersive training environments were compared with traditional fire drills to assess differences in cognitive load, engagement, and stress levels, with findings validated through real-world testing [49].

Integrating IoT data and LLM into DTs introduces unique user interface (UI) and user experience (UX) design challenges, particularly in immersive VR environments. None of the platforms the team evaluated (Unreal, NVIDIA Omniverse, and Autodesk Tandem) provided native support for embedding IoT dashboards or chatbot interfaces in VR. This lack of VR integration limits real-time interaction and data manipulation in VR.

Nevertheless, the team implemented two-way data exchange, enabling real-time control of systems such as machines and lighting through DT. Integrating smart and legacy devices into a unified platform fosters operational agility. However, translating these data-exchange protocols into VR remains difficult due to the limitations of motion controllers and hand-gesture inputs, which are less precise than traditional 2D interfaces like sliders or buttons.

4.2. Augmented Reality for Digital Twins

Augmented Reality (AR) offers a compelling alternative for DT visualization, particularly in construction and field applications. AR devices like Microsoft HoloLens can overlay BIM data directly onto real-world environments, enabling real-time inspection, coordination, and progress tracking. These overlays provide intuitive, context-aware visualizations that improve decision-making on-site. In addition to headsets, AR experiences can be delivered via smartphones and tablets for broader accessibility. Our team has implemented AR systems on mobile devices using image and 3D object recognition to trigger dynamic, scenario-specific information displays, and on HoloLens to assist digital fabrication [50]. While AR is gaining traction in construction management, its adoption within traditional DT workflows remains relatively limited.

4.3. Virtual Reality for Metaverse

XR is central to the implementation of the Metaverse, delivering the immersive, interactive qualities that define the medium. Metaverse platforms often incorporate multiplayer environments, AI-driven NPCs, and spatially responsive agents to facilitate collaboration, storytelling, and simulation. Platforms such as Meta Horizon, Mesh, VR Chat, Meeting Room, and Glue demonstrate the power of VR for multi-user engagement through avatars. Some systems even support naturalistic interaction using facial expression tracking, hand gestures, and motion capture, enhancing immersion beyond traditional conferencing tools.

Unlike DTs, metaverse environments are not bound by physical constraints. Circulation is often not limited by walls or gravity, enabling new spatial paradigms. In our design studio, students explored navigation modes including teleportation, zero-gravity movement, and multi-dimensional paths. These studio works revealed the importance of balancing creative freedom with intuitive spatial logic to support user orientation and memory (Previous Figure 1). Architectural design in the Metaverse demands a hybrid of functionality, aesthetic expression, and UX design. Students created virtual structures that ranged from hyper-realistic to surreal, unconstrained by material limitations or physical laws.

4.4. Augmented Reality for Metaverse

Though AR holds a significant promise for blending metaverse content with the physical world, demonstrated by successes like Pokémon Go, its application in immersive Metaverse platforms remains underdeveloped. Our evaluation of metaverse platforms, including Virbela, Glue, Microsoft Mesh, Mozilla Hubs, and Hyperspace, revealed that most of them lack robust AR capabilities, focusing instead on screen or VR-based experiences.

However, as wearable AR devices like Google Glasses and Meta Glasses advance, the convergence of spatial computing and the Metaverse is expected to accelerate. This evolution could enable truly hybrid experiences where digital content interacts fluidly with physical environments, offering context-aware, immersive engagements that blur the boundary between real and virtual worlds of the Metaverse.

5. Conclusions: Convergence of DT and Metaverse

Our research team has investigated both DT and Metaverse applications, employing AI and XR technologies to delineate the boundaries between these frequently conflated concepts. Through our analysis, we have identified key differences in their implementation, development, and management approaches. However, technological convergence is becoming increasingly evident, driven largely by advancements in AI and XR. For example, a virtual building can operate simultaneously as a DT—incorporating real-time IoT and BIM data—and as an immersive, persistent Metaverse environment that facilitates social interaction and cultural engagement.

For instance, using VR headsets, users can navigate hybrid DT+Metaverse spaces, which include scenario-based simulations (e.g., fire, smoke, and emergency egress), path tracking, and eye-tracking to assess spatial behavior and improve training outcomes. These hybrid environments can also host virtual exhibitions that run concurrently with in-person events [51], effectively merging the functionalities of DT and the Metaverse (previous Figure 2). A recent example of this convergence is Zaha Hadid Architects’ development of a virtual London [52], blending mirrored reality with imagined worlds. As Jean Baudrillard depicts the three orders of simulacra [53], the convergence of DT and Metaverse can evolve into a new type of reality.

The successful integration of DTs and the Metaverse suggests a promising future for these technologies and their subsequent convergence. As DTs evolve, they will play a central role in optimizing operations, enhancing design processes, and fostering innovation across various industries. Incorporating machine learning further highlights the transformative potential of AI, supporting more efficient reality capture, augmenting decision-making processes, and improving user interaction within both environments.

However, current DT+Metaverse implementations face several challenges that hinder widespread adoption and efficiency. A major issue is the lack of standardized frameworks for data modeling [54], particularly the challenge of ensuring interoperability between BIM and IoT data sources [7]. Additionally, there is a notable absence of well-developed platforms that support the seamless integration of DTs and the Metaverse. Few studies have evaluated the performance or maturity of DT or Metaverse systems, with existing research focusing mainly on blockchain systems for performance-based smart contracts [55] or multi-objective optimization frameworks [56].

Furthermore, adoption gaps persist for leading DT platforms—including NVIDIA Omniverse and Autodesk Tandem—as well as Metaverse systems like Meta Horizon and Microsoft Mesh. While advanced engines such as Unreal Engine and Fortnite Creative provide robust Metaverse development tools (particularly for gaming), AEC researchers and developers continue to face critical challenges, including the high costs of reality capture, the need for custom UI development to visualize IoT data, the complexity of implementing bidirectional data exchange systems, and difficulties in effectively integrating LLMs to support complex interactions.

In conclusion, although DT-Metaverse convergence represents a transformative opportunity for technological progress, substantial barriers must be addressed to fully realize these systems’ potential in both operational efficiency and user experience domains.