INTELLECTUM: A Hybrid AR-VR Metaverse Framework for Smart Cities

Abstract

This work presents INTELLECTUM as a reference architecture and design-time evaluation framework for multi-entity XR–AI–digital twin systems. Rather than optimizing a specific implementation, the paper formalizes architectural invariants, event semantics, and coordination mechanisms that precede and inform system realization. INTELLECTUM provides a conceptual framework for structuring interactions across physical and virtual environments, emphasizing human-centered design, immersive digital twins, and collaborative extended-reality workspaces. The technical specification defines core architectural components, human integration modalities via WebXR and heterogeneous sensor networks, and representative usage scenarios within smart city ecosystems. By enabling AI-assisted urban planning, interactive simulation, and multi-actor coordination, INTELLECTUM positions itself as an XR-based architectural foundation for next-generation smart city platforms.

1. Introduction

Rapid urbanization and increasing digitalization of metropolitan environments have intensified the need for advanced human–machine interfaces that bridge physical and virtual systems in a coherent manner. Smart city (SC) initiatives increasingly rely on digital twins, immersive visualization, and artificial intelligence (AI) assisting decision support to manage complex urban infrastructures. However, existing platforms often fragment these capabilities across disconnected tools, lack formal coordination mechanisms between heterogeneous actors, and provide limited support for continuous interaction across reality–virtuality boundaries.

INTELLECTUM is introduced in this work as a conceptual architecture that addresses these challenges at the design level. Rather than proposing a specific implementation, the framework formalizes architectural principles for integrating extended reality (XR), artificial intelligence agents, the Internet of Things (IoT), and digital twins (DTs) into a unified interaction environment. The focus is on defining event semantics, coordination logic, and interaction invariants that enable consistent behavior across physical and virtual contexts, independent of underlying hardware or deployment choices.

A continuous reality–virtuality spectrum allows human participants to smoothly transition between augmented, mixed, and fully virtual representations of the same urban environment. Unlike conventional XR systems that treat AR and VR as separate modes, the architecture conceptualizes immersion as a controllable parameter within a unified interaction model. This design supports continuity of perception, spatial orientation, and task context, which is essential for urban planning, infrastructure monitoring, and collaborative decision-making scenarios. Shared environments also support transparent scenario exploration and participatory decision-making among municipal stakeholders in urban digital twins (UDTs).

Beyond human interaction, the framework formalizes multi-entity coordination across humans, AI agents, and embodied systems such as robots or sensor-equipped infrastructure components. By abstracting interaction through a common event-driven framework, the architecture enables heterogeneous actors to operate within a shared spatiotemporal model of the city. This approach supports collaborative workflows in which human intent, automated reasoning, and environmental data jointly influence system behavior.

The objective of this work is architectural soundness rather than empirical optimization. INTELLECTUM integrates heterogeneous subsystems (XR, AI, UDT, and IoT) whose full operational integration across distributed physical deployments represents a multi-year engineering effort beyond the scope of a single research prototype. The architecture spans multiple coordination boundaries: real-time perceptual rendering in XR clients, event-driven synchronization through WebRTC/WebSocket channels, deterministic state evolution in the Event Store, asynchronous agent inference loops, and physical sensor feedback from robotic inspectors (Figure 1). Validating these interactions empirically would require simultaneous deployment of all subsystems under production-scale conditions, which is infeasible at the architectural design stage.

Consequently, evaluation is conducted at design time through formal validation of architectural properties rather than runtime benchmarking. Design-time evaluation assesses whether the proposed architecture satisfies its stated requirements under formal interaction models, structural constraints, and complexity bounds. This approach is familiar in distributed systems and reference architecture research, where feasibility, consistency, and scalability are validated prior to implementation [1,2].

2. Related Works

2.1. Public Urban Digital Twins

DTs represent next-generation SC architectures by integrating planning, management, and service delivery within a unified computational framework. Their application spans urban development, infrastructure coordination, and public service optimization [3]. UDTs are not neutral technical artifacts—they reflect underlying governance objectives, planning paradigms, and institutional priorities. Without explicitly aligning social, political, and technical dimensions, deployment of DT may risk internal inconsistency and limited policy relevance in urban management contexts [4].

SC concepts can be applied within municipal infrastructure management by embedding digital monitoring, analytics, and coordination tools directly into existing engineering workflows. Such integration supports policy-relevant decision-making by linking technical system states with asset management and long-term investment planning [5].

Recent research in UDTs has increasingly incorporated immersive technologies to move beyond static visualization toward interactive, human-centered decision-making environments. Luo et al. [6] present a perception-powered UDT that integrates photo-realistic rendering with immersive virtual reality to capture and predict subjective human perceptions such as safety and liveliness in simulated urban scenarios. By combining semantic segmentation of urban imagery with VR-based perception experiments, their work demonstrates how human perceptual feedback can be systematically embedded into a DT pipeline. While the focus is primarily on urban planning and sustainability, the study illustrates a shift toward treating the DT as an interactive spatiotemporal model in which human perception becomes an operational input rather than an external evaluation layer.

DTs constitute a parallel research stream relevant to engineering-oriented metaverse systems. Mazzetto [7] reviews UDT integration and highlights their importance for SC management, sustainability, and predictive urban analytics. Similarly, another bibliometric study of over 4200 DT-related papers identified core themes in IoT-enabled simulation, real-time data fusion, and urban infrastructure modeling [8], while these DT studies emphasize large-scale environmental and infrastructural modeling, they generally lack mechanisms for deterministic multi-agent collaboration or decentralized and trusted contributions.

DT cities represent a shift toward data-driven models of urban governance, where real-time data streams, simulation, and automation inform policy coordination. The convergence of AI, IoT, and blockchain technologies enables new governance mechanisms but also introduces complexity in accountability and system oversight [9]. Integration of IoT and agents under UDT is particularly important for adaptive urban systems, requiring solutions to ensure data fusion, interoperatibility, and real-time consistency for agent–robot–human coordination across city-scale environments [10]. Recent research demonstrates that coordinated DT ecosystems improve interoperability, hotspot prediction, and eco-driving assistance in metropolitan environments [11].

XR technologies can broaden public participation in urban planning by enabling immersive visualization and remote engagement. Such tools improve transparency and facilitate feedback collection, but their effectiveness depends on usability, accessibility, and integration with existing participatory processes [12]. Web-based XR solutions demonstrate potential for professional urban planning applications by improving accessibility and reducing hardware barriers. Adoption is strongly influenced by interface familiarity and alignment with established planning practices [13].

2.2. XR/VR and Shared Spatiotemporal Interaction

Integration of AR experience with robotics expands possibilities for industrial and urban tasks. Qian et al. [14] provided a comprehensive review of AR integration in robotic-assisted surgery. Their survey analyzed hardware architectures, application paradigms, and clinical findings showing how AR can enhance surgical precision, reduce visual distraction, and improve situational awareness in minimally invasive procedures. The authors also discuss challenges such as visual clutter and propose mechanisms to ensure safe and efficient AR-based interfaces. This demonstrates how AR can already enhance precision in various contexts, being adaptable also to urban environments.

Diehl et al. [15] introduced an AR interface that enables humans to verify robot learning outcomes prior to execution. The system embedded a virtual simulation directly into the physical environment and coupled it with semantic action information, allowing users to assess whether the robot had acquired a task correctly. The authors compared three visualization designs, including AR-enhanced and traditional RViz-based interfaces, and conducted a user study with 18 participants. Results displayed that AR enhances realistic feedback, improves error detection performance, and reduces the need for environment modeling, highlighting the advantages of AR for human–robot collaboration.

Beyond planning-oriented applications, XR-enhanced DTs have been proposed as coordination environments where humans and autonomous agents jointly operate. Wang et al. [16] introduce XR-DT, an extended reality–enhanced DT framework for agentic mobile robots that tightly couples real-time sensor data, simulated environments, and human feedback through AR, VR, and MR interfaces. Their hierarchical architecture enables bidirectional understanding between humans and robots, allowing both to reason over a shared spatiotemporal representation of the environment. Importantly, the framework treats human interactions, robot actions, and environmental dynamics as co-evolving elements within the same DT, thereby supporting interpretable and trustworthy human–robot interaction in dynamic, shared spaces.

Complementing system- and agent-oriented perspectives, recent work on embodied interaction in DTs highlights the importance of precise spatial and temporal alignment for social and collaborative experiences. Zhang et al. [17] extend the DT paradigm to location-based social VR by integrating full-body avatars, hand tracking, and synchronized physical sensations. Their results show that accurate alignment between physical and virtual environments enables natural social gestures and interactions, fostering trust and presence among participants. Although focused on social VR, this work reinforces the idea that DTs can serve as shared spatiotemporal spaces where humans and AI can coordinate actions and perceptions in real time.

2.3. Human Integration Technical Specification

Immersion and transparency remain central challenges in AR-enhanced robotic applications. Although AR can significantly improve visual feedback and situational awareness, the perception is still constrained by the field of view (FoV), whether using a monitor, head-mounted display, or projection system. A limited FoV restricts the amount of contextual information available, potentially reducing immersion and impairing the operator’s ability to interpret robot behavior or environmental cues. Therefore, achieving effective AR–robot integration requires careful design of visual overlays, adaptive rendering strategies, and interface techniques that mitigate FoV limitations while preserving clarity and operational transparency [18].

The role of XR/VR in integrating humans into adaptive decision loops is further explored by Yigitbas et al. [19], who investigate human-in-the-loop adaptive systems using DTs coupled with immersive VR interfaces. Through procedural and mixed-initiative control strategies, humans can either directly intervene in system adaptations or specify goal states that are resolved by automated planners. This approach demonstrates how XR-based DTs can function as shared operational models where human intent and machine reasoning interact within a unified temporal and spatial context.

2.4. Metaverse Modalities

Recent research on metaverse systems, DTs, and cyber–physical integration reveals a rapidly evolving ecosystem that spans XR interfaces, real-time data infrastructures, AI agents, and decentralized computation layers. A recent large-scale survey by Rawat et al. [20] provides an extensive overview of metaverse fundamentals, highlighting its architectural requirements, emerging standards, enabling technologies (XR, AI, and ubiquitous computing), and cross-domain applications. Their work underscores the need for architectures capable of supporting multimodal inputs, distributed computation, and complex cyber–physical capabilities for interactions.

From the perspective of real-time metaverse environments, Hatami et al. [21] analyze the technological foundations required for low-latency, high-fidelity virtual worlds. Identified challenges involved synchronization delays, sensor–network bottlenecks, and interoperability limitations between digital and physical systems.

The vision in conventional social metaverse (CSM) focuses primarily on social presence, embodiment, and avatar-centric interaction, coupled with infrastructure that relies on centralized cloud rendering, actor replication, and proprietary XR pipelines [22]. This results in strong performance for social spaces but limited guarantees for interaction determinism, verifiable state evolution, or persistent multi-entity coordination. In contrast, INTELLECTUM is designed as a coordination-centric XR environment, emphasizing deterministic event synchronization, multi-entity interaction governance, and adaptive reality–virtuality blending.

2.5. Privacy in Real-Time Applications

Recent literature highlights that security and privacy remain critical barriers to the deployment of large-scale XR-enabled SC platforms. Braun et al. [23] demonstrated that SCs introduce uniquely high-dimensional data streams, which significantly amplify the privacy attack surface. Their analysis identifies five fundamental vulnerabilities: privacy leakage from sensor-rich environments, large-scale network exposure, untrustful data exchange, misuse of AI analytics, and cascading failures in interconnected systems.

On security and system integrity, Wang et al. [24] provide a detailed survey of metaverse security and privacy challenges, noting issues such as identity spoofing, cross-domain data tampering, and the absence of verifiable synchronization between virtual and physical worlds.

Client-side local re-projection and time-warp techniques aim to capture previously received update frames in accordance with the most recent tracking or state information (display refresh), thus bridging the temporal gap when remote updates arrive late. The time warp represents an extended VR experience in which motion-to-photon (MTP) delays are reduced by corrections for the optical aberration of VR lenses and transformation of the stereoscopic rendering from real-time head tracking information to the correct view at applicable display [25]. The time warp has been applied to increase the perceived frame rate and their consistency in asynchronous stereoscopic rendering, allowing optimized hardware usage as rendering with the display refresh cycle does not require direct synchronization [25].

Real-time communication (RTC) systems use an ABR algorithm to achieve minimal delays and maximum quality. RTC adjusts video bitrate dynamically based on available network bandwidth. Since issues typically arise from the conflict between the network’s bitrate targets and codec-based bitrate control, Li et al. [26] proposed the utilization of multi-agent reinforcement learning with multi-dimensional ABR algorithm to provide enhanced quality of experience (QoE). Their approach seeks adaptive behavior with collectively adjusted encoding factors, such as resolution, state-based frame rates, and quantization parameters. These pre-configurations established low latencies without compromising quality in peer-to-peer contexts by dynamically adjusting encoded parameters based on available bandwidth and video feedback [26].

Dead reckoning allows robotic entities, AI agents, and virtual avatars to maintain plausible continuity of motion even during packet loss or delayed updates. It is applied in distributed applications that require controlled bandwidth usage and latency [27,28]. Simulations in distributed virtual environments (DVEs) and online gaming have particularly drawn interest. Dead reckoning methods typically apply kinematic models directly for movement estimates, but further optimizations have been obtained by extrapolating environmental variables and human interactions [28]. Solutions for dead reckoning modeling can touch different contexts. Lin et al. [29] introduced multinomial logit models (MLM) with predict function sets for image smoothness, establishing discrete decision boundaries based on smoothing steps. MLM was applied to aerospace applications but also sets one set of tools for DVEs in highly diverse contexts.

A 2025 taxonomy-based review by Antuley et al. [30] emphasizes that future SC ecosystems will require hybrid authentication and authorization models capable of spanning AI-driven services, blockchain-secured infrastructures, and quantum-resilient communication layers. Their work highlights the growing demand for frameworks that guarantee confidentiality, integrity, availability, and trust in distributed urban infrastructures. From a governance perspective, Barcik et al. [31] show that the digitalization of cities requires robust mechanisms to protect critical urban services from cyberattacks, misinformation streams, and systemic disruptions. Findings underscore that DTs expand the security footprint by mirroring real infrastructure, making the virtual model itself a target.

Latency-related security vulnerabilities have also been highlighted in recent XR research. Monaco et al. [32] demonstrate that predictive latency models can substantially reduce user-perceived delay in distributed interactive systems such as cloud gaming. Their machine-learning-based approach forecasts network conditions to mitigate spikes and concept drift. Peter and Yazdanian [33] apply pose-prediction methods to compensate for network delays in offloaded XR computations, ensuring that user movements remain synchronized with remotely rendered content. INTELLECTUM integrates these latency-compensation techniques (dead reckoning, adaptive bitrate control, and predictive rendering) to maintain temporal stability and reduce attack vectors that exploit timing inconsistencies.

3. INTELLECTUM Interaction Pipelines

The protocols introduced in the following sections describe logical interaction flows rather than low-level implementations. The interaction protocols are closer to state-transition systems than numerical algorithms. Data structures, serialization formats, and failure recovery are implementation-dependent and outside the scope of this architectural specification.

3.1. Cognitive Environment

The cognitive environment necessitates context-aware data between entities with sufficient precision. High precision is required, especially in non-VR states where actuated AI entities or robots are integrated into real urban environments. However, network capacity may become a bottleneck if all raw sensor data are processed.

One approach for optimization is to apply reinforcement learning to establish a dynamic mode switch between lightweight updates and high-fidelity sensory sharing [34]. This is particularly applicable for AI agents, which could recognize and prioritize messages based on urgency. For example, concise status and event messaging over the global or regional network provides a persistent record that can be handled with lightweight updates. In high-urgency situations (e.g., emergency maneuvers), agents switch to high-fidelity mode to provide immediate response with rich sensory data. To ensure system integrity and trust, each agent has a reputation score that defines its access to high-fidelity communications. Since the agent interacts in a 3D workspace, its positions and event locations should be mapped in order to estimate its state at a specific time. Following previous research [34], the state for the agent at time t can be defined as (Equation (1)):

$$s_t=\{p\left(t\right),T\left(t\right),E\left(t\right),L\left(t\right)\}$$

(1)

where $p\left(t\right)$ is the position and proximity to other agents or entities, $T\left(t\right)$ expresses the active task, $E\left(t\right)$ is an event indicator, and $L\left(t\right)$ specifies the computational load in the current network.

Algorithm 1 lays the high-level interaction pipeline for interactive cross-communications for cognitive systems. Implementation-specific concerns such as internal data structures, message serialization, synchronization primitives, error handling, and failure recovery are delegated to the underlying runtime and network middleware and are therefore omitted for clarity.

The cognitive environment seeks to integrate both symbolic and probabilistic components in hybrid AI reasoning, which influences the formalization of the trust system. Intelligent agents operating within this layer rely on a dual inference pipeline where a symbolic rule base enforces deterministic constraints and an LLM-driven probabilistic reasoner provides flexible contextual inference. To ensure that agent decisions remain both adaptive and verifiable, a hybrid trust model is employed to evaluate the correctness of LLM-generated outputs against symbolic constraints. Proposition 1 does not assert logical completeness or probabilistic optimality. It provides a practical evaluation model under the assumption that symbolic constraints are sound and that LLM confidence estimates are calibrated.

Proposition 1

(Hybrid AI Trust Evaluation Model). Assume the following conditions hold:

1. 

Let a hybrid agent combine a symbolic rule base $\mathcal{R}$ with a probabilistic LLM reasoner $\mathcal{L}$.

2. 

Given an inference result x proposed by $\mathcal{L}$, its correctness probability under the hybrid model is:

$$\mathrm{Pr}\left[x\mathit{is}\mathit{correct}\right]=\lambda ·\mathrm{Pr}[x\vDash \mathcal{R}]+(1-\lambda )·C_{\mathcal{L}},$$

(2)

where $\mathrm{Pr}[x\vDash \mathcal{R}]\in \{0,1\}$ indicates whether the symbolic rule base verifies the inference, $C_{\mathcal{L}}$ is the LLM confidence score, and $\lambda \in [0,1]$ is the symbolic grounding coefficient.

3. 

Let trustworthy condition be

$$\mathrm{Pr}\left[x\mathit{is}\mathit{correct}\right]\ge \tau ,$$

(3)

for a trust threshold τ, the system accepts the inference as trustworthy, providing a design-time trust evaluation criterion under explicit assumptions for hybrid symbolic–LLM reasoning.

Algorithm 1 INTELLECTUM Cognitive Environment Interaction Protocol

1: procedure XRInteractionLoop   2:     $evt\leftarrow \mathrm{CaptureXRInput}\left(\right)$    ▹ Head pose, controllers, gesture, camera frame   3:     $\mathrm{SDK}.\mathrm{SendEvent}\left(evt\right)$            ▹ Forward to SDK-JS layer   4: end procedure   5: procedure SDK.SendEvent($evt$)   6:     if $evt.type=\mathrm{SceneUpdate}$ then   7:         $\mathrm{SyncLayer}.\mathrm{Broadcast}\left(evt\right)$          ▹ WebRTC SFU → all peers   8:     else if $evt.type=\mathrm{ModelOp}$ then   9:         $\mathrm{Store}(evt.asset)$                   ▹ Asset persistence 10:         $\mathrm{Runtime}.\mathrm{Notify}\left(evt\right)$              ▹ Inform agent runtime 11:     else if $evt.type=\mathrm{UserAction}$ then 12:         $\mathrm{Runtime}.\mathrm{QueryAgent}\left(evt\right)$         ▹ Interactive cognition request 13:     end if 14: end procedure 15: procedure SyncLayer.Broadcast($evt$) 16:     $session\leftarrow \mathrm{SessionManager}.\mathrm{Resolve}(evt.user)$ 17:     $\mathrm{WebRTC}.\mathrm{Send}(session.channels,evt)$     ▹ Propagate via data channels 18: end procedure 19: procedure Runtime.QueryAgent($evt$) 20:     $context\leftarrow \mathrm{ContextBuilder}.\mathrm{Collect}\left(evt\right)$      ▹ Build multimodal context 21:     $m\leftarrow \mathrm{AI}.\mathrm{Retrieve}\left(context\right)$                 ▹ Associative recall 22:     $a\leftarrow \mathrm{AI}.\mathrm{Reason}(m,context)$              ▹ Continual inference 23:     $\mathrm{Runtime}.\mathrm{Emit}\left(a\right)$ 24: end procedure 25: procedure Runtime.Emit(a) 26:     if $a.type=\mathrm{SceneChange}$ then 27:         $\mathrm{SDK}.\mathrm{EmitToClient}\left(a\right)$          ▹ Visual updates to XR client 28:     else if $a.type=\mathrm{AgentAction}$ then 29:         $\mathrm{SyncLayer}.\mathrm{Broadcast}\left(a\right)$          ▹ Multi-entity propagation 30:     else if $a.type=\mathrm{Store}$ then 31:         $\mathrm{Store}(a.payload)$                ▹ Persist model output 32:     end if 33: end procedure 34: procedure SDK.EmitToClient($msg$) 35:     $\mathrm{XRClient}.\mathrm{Apply}\left(msg\right)$         ▹ Modify scene, overlay, avatar, UI 36: end procedure

3.2. Reality–Virtuality Continuum

While Algorithm 1 describes the cognitive event-processing pipeline that governs communication between the XR Client, software development kit (SDK), synchronization layer, and agent runtime, it abstracts away the details of how the visual environment is actually composed. INTELLECTUM separates these concerns by using the cognitive environment that manages what events occur and which entities respond, whereas the reality–virtuality subsystem determines how the world is rendered for the user. The transition coefficient $\gamma$ therefore operates orthogonally to the cognitive loop—events propagate identically regardless of immersion level, but the perceptual outcome is blended according to the user’s position along the reality–virtuality continuum. Algorithm 2 formalizes this rendering-specific process and shows how physical and virtual layers are composited in real time. The blending mechanism in Algorithm 2 activates Equation (4) by directly integrating the reality coefficient $\gamma$ into the rendering pipeline.

Algorithm 2 Reality–Virtuality Blending Protocol in INTELLECTUM

1: $\gamma \leftarrow \mathrm{InitRealityCoefficient}\left(\right)$   ▹ 0 = VR (fully virtual), 1 = physical passthrough   2: procedure HandleXRInput   3:     $evt\leftarrow \mathrm{CaptureXRInput}\left(\right)$       ▹ User gestures, headset passthrough, sliders   4:     if $evt.type=\mathrm{AdjustGamma}$ then   5:         $\gamma \leftarrow evt.value$               ▹ User/device modifies AR–VR blend   6:     end if   7:     $\mathrm{UpdateRealityBlend}\left(\gamma \right)$          ▹ Recompute rendered environment   8: end procedure   9: procedure UpdateRealityBlend($\gamma$) 10:     $P\leftarrow \mathrm{CapturePhysicalFeed}\left(\right)$       ▹ Camera or passthrough feed as texture 11:     $V\leftarrow \mathrm{RenderVirtualScene}\left(\right)$           ▹ 3D assets, UI layers, virtual entities 12:     $R\leftarrow \gamma ·P+(1-\gamma )·V$       ▹ Reality–virtuality blending (Equation (4)) 13:     $\mathrm{Display}\left(R\right)$                      ▹ Final composited frame 14: end procedure 15: procedure AgentInfluence 16:     $a\leftarrow \mathrm{AgentOutput}\left(\right)$       ▹ AI retrieval and reasoning may adjust immersion 17:     if $a.type=\mathrm{AdjustGamma}$ then 18:         $\gamma \leftarrow a.value$                 ▹ Agent suggests immersion level change 19:         $\mathrm{UpdateRealityBlend}\left(\gamma \right)$ 20:     end if 21: end procedure

Equation (2) is designed to provide a seamless transition mechanism between reality states through a central toggle controller that manages rendering pipelines, interaction modalities, and data flow based on user positioning along the reality–virtuality spectrum. The continuum is mathematically represented as (Equation (4)):

$$R=(\gamma -1)P+\gamma V$$

(4)

where R is the rendered composite environment, representing the blending ratio between physical and virtual content, with 1 presenting virtual dominance. $\gamma$ is the virtuality coefficient (a scalar in $[0,1]$) ranging from 0 (fully physical) to 1 (fully virtual). V and P are the virtual and physical render buffers, respectively. Physical components include video textures, passthrough XR, depth occlusion, and real-time motion tracking. Virtual components contain 3D models, overlays, holograms, effects, digital avatars, and scene lighting. Four operational regions are recognizable for the rendered environment:

• Enhanced Reality (ER): Predominantly physical environment with minimal virtual overlays ($\gamma \le 0.3$). ER with a low virtual coefficient is applicable for navigation and overlay displays.
• Mixed Reality (MR): Balanced physical–virtual composition ($0.3<\gamma <0.7$). MR has an evenly blended virtual coefficient level, providing deeper interactions for tasks such as training and design.
• Augmented Reality (AR): Predominantly virtual environment with residual physical grounding ($0.7\le \gamma <1.0$). AR is dominantly virtual, ideal for fast prototyping and simulations.
• Virtual Reality (VR): Fully virtual environment ($\gamma =1.0$). VR is a completely virtual state for broadcasting events and DTs.

While $\gamma$ is a continuous variable, categorical regions ER, MR, AR, and VR are considered as semantic abstractions between different ranges for policy selection, rendering pipelines, and interaction constraints. Device capability influences the maximum achievable reality coefficient $\gamma$ within the reality–virtuality continuum. Low-end devices maintain higher reliance on physical passthrough ($\gamma \approx 1$), while professional XR systems can reliably sustain near-virtual scenes ($\gamma \to 0$) without compromising frame rate or spatial coherence. This adaptive control ensures uniform experience quality across various user hardware.

Seamless reality switching during adaptations to different multi-modal tasks is assigned to the reality switch (Equation (5)), which is defined as:

$$D{T}_{urban}=\sum _{i=1}^n{S}_i·\Phi \left(I_i\right)+\lambda ·\mathrm{RealitySwitch}(AR,VR)$$

(5)

where $D{T}_{urban}$ represents the current rendered state of the UDT, $I_i$ denotes raw input from the i-th sensing modality from input streaming devices, $S_i$ is a weighting or activation coefficient reflecting the relevance or reliability of that sensor stream, and $\Phi (·)$ is a modality-specific transformation function that maps raw sensor inputs into a unified DT representation involving actions such as feature extraction, spatial alignment, or semantic encoding. The term $\mathrm{RealitySwitch}(AR,VR)$ represents a control function governing transitions between augmented and virtual rendering modes, while the scalar coefficient $\lambda$ regulates the influence of perceptual mode switching relative to sensor-driven updates.

Depending on device capabilities (such as camera passthrough availability, WebXR subsystem support, or desktop camera feeds), the reality coefficient $\gamma$ (float between 0 and 1) is dynamically adjusted to reflect the user’s perceptual state. A value of $\gamma =0$ corresponds to fully virtual VR environments, whereas $\gamma >0$ introduces increasing contributions of the physical feed, enabling ER and MR experiences. Because this protocol is decoupled from cognitive routing, INTELLECTUM supports seamless transitions between AR and VR modes, continuous reality blending during agent interaction, and adaptive immersion levels that can be modulated by user input, device sensors, or higher-level agent reasoning. This establishes a unified and extensible perceptual layer that integrates physical and digital worlds.

3.3. Multi-Entity Integration Framework

Multi-entity integration framework is organized using the synchronization layer, the core runtime responsible for real-time message propagation, state harmonization, and cross-entity coordination. It serves as the canonical interface through which all agents, robots, and humans exchange state updates.

The synchronization layer provides three fundamental services: (i) temporal synchronization of distributed entities, (ii) session-level consistency through WebRTC channels and verifiable state diffs, and (iii) event propagation toward persistence (data infrastructure layer). Through this mechanism, the synchronization layer acts as the arbitration point that guarantees causality, ordering, and reproducibility of multi-entity interactions across diverse environments.

Algorithm 3 for interactions is executed within the synchronization layer, which functions as the authoritative coordination point for multi-entity communication. Upon permission validation, the synchronization layer establishes a session channel between the entities, ensuring that interaction events follow the system’s global ordering constraints. The synchronization layer logs the interaction, updates both local and distributed entity states, and disseminates the resulting changes to all subscribed peers.

A unified synchronization core guarantees secure, deterministic, and extensible multi-entity coordination across agents, robots, and human users. The synchronization layer functions as the authoritative gateway for all cross-entity communication, enforcing global ordering, validating permissions, and propagating authenticated state transitions. Each interaction request is validated through lightweight cryptographic checks, ensuring that entity identities, session keys, and capabilities are verified before any state mutation occurs. This establishes a trust mechanism across distributed XR, robotic, and AI-driven environments.

Algorithm 3 Multi-Entity Interaction Protocol

1: procedure EntityInteraction($entit{y}_1,entit{y}_2,interaction\_type$) 2:     $auth\leftarrow \mathrm{VerifyPermissions}(entit{y}_1,entit{y}_2)$ 3:     if $auth=True$ then 4:         $session\leftarrow \mathrm{CreateInteractionSession}(entit{y}_1,entit{y}_2)$ 5:         $\mathrm{LogInteraction}(session,interaction\_type)$ 6:         $\mathrm{UpdateEntityStates}(entit{y}_1,entit{y}_2)$ 7:         $\mathrm{BroadcastStateChange}(entit{y}_1,entit{y}_2)$ 8:     end if 9: end procedure

Synchronization is achieved through a hybrid real-time pipeline where WebRTC data channels are combined with session-level consistency guarantees. The synchronization layer broadcasts entity updates using timestamped state diffs, enabling precise temporal alignment between participants while preventing race conditions and divergent world states. To maintain deterministic replay and conflict-free state reconstruction, the synchronization layer archives all events to the data infrastructure and selectively anchors critical state commitments to the persistent and event-based storage. This upward and downward propagation ensures coherence across volatile, semi-persistent, and immutable storage domains. Consistency of multi-entity synchronization is ensured when (Equation  (6)):

$${\tau }_i<{\tau }_j\Rightarrow {\mathbf{S}}_i\prec {\mathbf{S}}_j$$

(6)

The cross-entity trust score is computed as:

$$T_i=\alpha ·R_i+(1-\alpha )·H_i$$

(7)

where $R_i$ represents the entity’s reputation and $H_i$ provides historical consistency and compliance. Trust score blending parameter $\alpha$ balances the influence of recent reputation versus historical consistency when computing an entity’s overall trust score $T_i$.

Extensibility is embedded into the framework through modular interfaces that expose standardized protocols for introducing new entity types, external systems, or behavioral modules. Interoperability with robotics ecosystems, IoT networks, and simulation pipelines is achieved through the interoperability layer, which consumes synchronized state events and translates them into external semantics. Similarly, intelligent agents and human XR clients can be extended with new sensing modalities, behavioral policies, or learning components without modifying the synchronization core.

To formalize cross-entity coordination, we define each entity $e_i$ by a state vector:

$${\mathbf{s}}_i\left(t\right)=\{p_i\left(t\right),v_i\left(t\right),c_i\left(t\right),{\theta }_i\left(t\right)\}$$

(8)

where $p_i$ is position, $v_i$ velocity, $c_i$ capability vector, and ${\theta }_i$ cognitive (or intentional) state. The synchronization layer computes a global scene state:

$$\mathbf{S}\left(t\right)=\bigcup _{i=1}^N{\mathbf{s}}_i\left(t\right)$$

(9)

All interaction events form a time-ordered log:

$$\mathcal{L}=\left[(E_1,{\tau }_1,a_1),(E_2,{\tau }_2,a_2),\dots ,(E_k,{\tau }_k,a_k),\dots \right]$$

(10)

where $E_k\subseteq \{e_1,\dots ,e_N\}$ is the set of entities participating in the k-th event (allowing unary, pairwise, or multi-entity interactions), ${\tau }_k$ is the Lamport timestamp, and $a_k$ is the interaction action.

The causality condition in Equation (6) ensures that all clients and agents interpret events in the same logical order, even if network latency causes different physical arrival sequences. When the synchronization layer receives updates, it applies a deterministic merge operator that reconstructs the world state based on Equation (9) precedence, guaranteeing that every participant converges toward the same global scene regardless of timing variations or packet loss.

The integration of trust scoring (Equation (7)) further refines state interpretation by weighting updates based on the reputation and historical reliability of each entity. This becomes essential in environments where physical robots, IoT devices, human operators, and autonomous agents contribute simultaneously to the evolving world state. Entities with low consistency or repeated contradictory updates can be automatically down-weighted, while reliable entities gain stronger influence on the synchronized global state $\mathbf{S}\left(t\right)$. Proposition 2 states that coordination remains polynomial assuming bounded entity state size and fixed merge operators, not as a universal complexity claim.

Proposition 2

(Bounded Coordination Complexity). Assume the following:

1. 

There are n active entities and m events in the Lamport log $\mathcal{L}$.

2. 

State merging is performed using algorithm MergeState with complexity $O(n·f(n\left)\right)$, where $f\left(n\right)$ is the cost of merging individual entity states.

3. 

Lamport timestamp evaluation is performed for all events in $\mathcal{L}$ in $O\left(m\right)$ time.

4. 

Conflict-Free Replicated Data Type (CRDT) reconciliation for conflict-free updates has complexity $O(n·g(n\left)\right)$, where $g\left(n\right)$ is the cost of merging individual CRDT objects.

Then, the total decision-making time for any multi-entity interaction in INTELLECTUM satisfies:

$$T_{\mathrm{decision}}=O(n·f\left(n\right)+m+n·g\left(n\right)).$$

If $f\left(n\right)$ and $g\left(n\right)$ are polynomial, the overall complexity is polynomial in n and m.

Proposition 2 represents polynomial-time bounded coordination for all synchronized multi-entity interactions. Because all interaction events are recorded in the log (Equation (10)), the presented architecture can perform deterministic replay, anomaly detection, and post hoc verification. The log serves as a causal trace of the entire system’s evolution, enabling external systems to reconstruct past states using the same deterministic merge rules as the live system.

Overall, this design supports coordinated reasoning, conflict-free state propagation, and extensible behavior across multi-entities. Whether an update originates from a robot, an XR client, an AI agent, or an IoT sensor, the same mathematical structure defines how it enters and influences the shared environment.

An actor is defined as an entity that can emit and conduct event-triggered synchronization. Actor abstraction involves human XR clients, AI agents, and robotics/IoT devices. Equation (11) formalizes the actor abstractions:

$$a_i:=\langle i{d}_i,{\mathbf{s}}_i\left(t\right),{\mathcal{C}}_i,{\mathcal{P}}_i\rangle$$

(11)

where $i{d}_i$ is a globally unique identifier, ${\mathbf{s}}_i\left(t\right)$ is the local state vector for coordination (Equation (8)), ${\mathcal{C}}_i$ is the capability set, and ${\mathcal{P}}_i$ is the trust profile. Additionally, the decoupling of actors can be formally defined as Equation (12):

$$\forall a_i,a_j\in A,i\ne j:e_i\perp e_j|\mathrm{sync}\left(\mathbf{S}\left(t\right)\right)$$

(12)

where $a_i,a_j\in A$ are distinct actors in the system, $e_i,e_j$ are the event streams emitted by actors $a_i$ and $a_j$, respectively, $\mathbf{S}\left(t\right)$ is the global synchronized scene state (Equation (9)), and $\mathrm{sync}\left(\mathbf{S}\right(t\left)\right)$ denotes the synchronization layer that merges events. ⊥ indicates that the event streams are independent given the synchronization process.

3.4. Latency Management

To compensate for delays that arise from network jitter, sensor processing, WebRTC transport, and rendering overhead, INTELLECTUM uses predictive motion modeling and time-warp rendering strategies, similar to latency-compensation methods described in recent XR research [32,35,36,37].

The predictive rendering function for maintaining synchronization demands is (Equation (13)):

$$P_{render}(t+\Delta t)=S_{current}+V·\Delta t+{\displaystyle \frac{1}{2}}A·{(\Delta t)}^2$$

(13)

where $P_{render}$ is the predicted render position of an entity, $S_{current}$ is the current state, V is the velocity vector, and A is the estimated acceleration extracted from inertial measurement unit (IMU) and positional stream data.

To mitigate network variability, adaptive bitrate control (ABR), client-side local re-projections and time warp techniques, dead-reckoning motion models, and frame-level interpolation can be applied.

4. System Architecture

Several decisions affected the architectural solutions. These were formalized as requirements: event-driven design, formal event taxonomy, architectural properties, and public domain considerations.

4.1. Requirements

The architecture is derived from a set of design-time requirements that constrain how interactions across humans, agents, robots, and DTs can be coordinated between reality states. These requirements act as preconditions that rule out monolithic, synchronous, or tightly coupled designs and motivate the event-driven architecture introduced in Section 4.2. Validation of interactions between intelligent and automatic entities across reality states, architecture of INTELLECTUM was established based on the following requirements:

• Integration of adaptive cognitive systems
  The architecture must support heterogeneous reasoning processes (human intent, AI inference, and automated control) without assuming a shared execution model or synchronized internal state. This precludes centralized control logic and necessitates event-based coordination.
• Controllable reality–virtuality continuum mechanism for dynamic state transitions in interface
  Reality transitions (ER-AR–MR–VR) must be dynamically adjustable at runtime without invalidating system state or coordination logic. This requires separation between authoritative state changes and view-dependent rendering updates.
• Multi-entity collaboration across heterogeneous actors
  Humans, AI agents, robots, and infrastructure components must participate as first-class entities in shared interactions, with no hard-coded role assumptions. The architecture must therefore abstract interaction through a uniform event model rather than direct method coupling.
• Asset tracking and state synchronization
  Distributed entities must observe a consistent view of shared assets and environments under partial failures and asynchronous communication, requiring explicit event ordering, replayability, and conflict-tolerant state reconciliation.

4.2. Event-Driven Design

Figure 1 introduces a hybrid event-driven architecture following Command–Query Responsibility Segregation (CQRS), where state mutation events and read-model synchronization are decoupled to ensure scalability and deterministic replay.

By construction, the event propagation graph induced by Figure 1 admits a strict partial order over components. All durable events are appended to an append-only log L with Lamport timestamps. Role separation uses CQRS, meaning that projections never emit commands, nor do renders mutate authoritative state. Ingress, domain, projection, and real-time signal events are stacked into non-overlapping classes with monotonic timestamp assignment, preventing any causal feedback loop. Consequently, the event flow graph is directed and acyclic. The system is designed to scale to any number of actors without requiring coupling between them. This is constrained by the event-driven architecture (Figure 1) and the actor abstraction formalism (Equation (11)).

4.3. Formal Event Taxonomy

A disjoint and complete event taxonomy is defined as $\mathcal{E}$, where each class encodes distinct semantic roles and routing constraints. Perceptual events affect only representation, state mutation events modify authoritative DT state, and governance events provide verifiable accountability without influencing perception. Formal event taxonomy is presented in

Table 1. Formal event taxonomy and temporal semantics in INTELLECTUM.

Event Class	Temporal Semantics	Scope	Formal Constraints
Perceptual Events ${\mathcal{E}}_P$	Synchronous	Client/Session	$\forall E\in {\mathcal{E}}_P:\left|E\right|=1\wedge E\notin {\mathcal{E}}_S\wedge E\nrightarrow {\mathcal{E}}_G$
Interaction Events ${\mathcal{E}}_I$	Synchronous	User/Agent	$\forall E\in {\mathcal{E}}_I:E\subseteq \{e_1,\dots ,e_N\}\wedge E\to ({\mathcal{E}}_C\cup {\mathcal{E}}_S)$
Cognitive Events ${\mathcal{E}}_C$	Asynchronous	Agent Runtime	$\forall e\in {\mathcal{E}}_C:\exists s\in {\mathcal{E}}_S\vee e\mathrm{is}\mathrm{advisory}$
State Mutation Events ${\mathcal{E}}_S$	Ordered	Global/Shared	$\forall E\in {\mathcal{E}}_S:\exists t_i\in \mathbb{T}\wedge E_i\prec E_{i+1}$
Physical Feedback Events ${\mathcal{E}}_F$	Asynchronous	Environment	$\forall e\in {\mathcal{E}}_F:e\to {\mathcal{E}}_C\wedge e\notin {\mathcal{E}}_P$
Governance Events ${\mathcal{E}}_G$	Asynchronous	Audit/Trust	$\forall E\in {\mathcal{E}}_S:\exists g\in {\mathcal{E}}_G\mathrm{s}.\mathrm{t}.E\prec g$

.

4.4. Architectural Properties

Architectural properties define structural and behavioral guarantees, including clear boundaries between components, enforcement of event ordering, and consistency of the global synchronized scene state $\mathbf{S}\left(t\right)$ under multi-entity interactions. Assume the following formal conditions hold:

• Let the global synchronized scene state be defined as $\mathbf{S}\left(t\right)$ (Equation (9)).
• Let all interaction events be organized into the Lamport-ordered log $\mathcal{L}$ (Equation (10)).
• Assume that the synchronization layer performs:

  (i)

  state-vector merging on $\mathbf{S}\left(t\right)$;

  (ii)

  causality and ordering checks on $\mathcal{L}$;

  (iii)

  trust-score and permission validation, each using polynomial algorithms.

• Then the total decision-making time for any multi-entity interaction in INTELLECTUM satisfies:

  $$T_{\mathrm{decision}}=O\left(n^c\right),$$

  (14)

  where n is the number of active entities and c is a constant determined by the combined complexity of state merging, Lamport timestamp evaluation $\mathcal{L}$, and CRDT-based reconciliation.

These bounds hold under standard assumptions of reliable session connectivity and bounded message delays, consistent with distributed interactive systems literature. INTELLECTUM does not model intelligence nor its own cognition—its primary goal is to model coherent events between entities.

4.5. Public Domain Considerations

Architectural adaptations for the public domain and UDTs were constrained to three core considerations. Generalization of conditions was defined for collaborative workspaces, adaptive public services, and privacy and surveillance management.

4.5.1. Collaborative Workspaces

Collaborative environments allow distributed teams to co-manipulate DT layers, annotate real-time sensor data, and coordinate urban interventions. Recent SC research indicates that coordinated multi-user DTs improve decision-making efficiency and reduce operational friction in complex urban tasks [11]. Collaboration efficiency is expressed (Equation (15)):

$$E_{collab}={\displaystyle \frac{{\sum }_{i=1}^n{T}_{success\left(i\right)}}{{\sum }_{i=1}^n{T}_{total\left(i\right)}}}·C_{sync}·Q_{data}$$

(15)

where $E_{collab}$ represents multi-user performance, $T_{success}$ denotes successful interactions or tasks, $C_{sync}$ is the synchronization quality between distributed users, and $Q_{data}$ is the fidelity of shared data streams.

4.5.2. Adaptive Public Services

Context-aware public services dynamically adapt using both real-world sensor readings and AR/VR state overlays. SC administrators may adjust traffic management, environmental controls, or emergency response based on combined physical and digital insights. DT research identifies these adaptive service loops as key drivers of next-generation municipal management systems [38]. Context-aware public services adapt based on reality states (Equation (16)):

$$S_{adapted}=\beta ·U_{pref}+(1-\beta )·C_{env}·\gamma \left(R_{state}\right)$$

(16)

where $S_{adapted}$ is the computed service output, $U_{pref}$ represents user or citizen preferences, $C_{env}$ is the environmental context extracted from sensors, and $\gamma \left(R_{state}\right)$ represents the reality-state from INTELLECTUM’s AR/VR continuum.

4.5.3. Privacy and Surveillance Management

To protect user identity and sensor-derived data, INTELLECTUM uses federated learning together with differential privacy. The effective privacy-preserving transformation is defined as (Equation (17)):

$$\mathcal{M}\left(x\right)=f\left(x\right)+\mathrm{Lap}\left({\displaystyle \frac{\Delta f}{ϵ}}\right)$$

(17)

where $\mathcal{M}\left(x\right)$ is the privacy-preserving output, $f\left(x\right)$ is the local function (model updates), $\Delta f$ is its sensitivity, and $ϵ$ is the privacy budget. Only the privatized updates are transmitted to the coordination layer, while raw sensor data remains on the user’s device.

5. Technical Evaluation

Technical evaluation assesses the feasibility, correctness, and performance characteristics of the architecture by combining formal design-time analysis (Section 5.1) with synthetic evaluation scenarios (Section 5.2).

5.1. Design-Time Evaluation

Design-time evaluation assesses architectural validity without executing a full system implementation. Rather than relying on empirical benchmarks, this evaluation examines whether the proposed architecture satisfies its stated requirements under formal interaction models, structural constraints, and complexity bounds. This approach is common in distributed systems and reference architecture research, where feasibility, consistency, and scalability are established prior to implementation.

Design-time evaluation is supported by the reference constructs summarized in

Table 2. Design-time evaluation criteria references and reference constructs.

Dimension	Criterion	Design-Time Validation Reference
Pipeline Coverage	Interaction classes	Canonical interaction scenarios described in Section 4 and illustrated in Figure 1. Perception treated and covered in Algorithm 2 and Equation (4).
	Event type completeness	Formal event categories defined in Section 4.1 and Algorithm 1
	Actor participation	Unified actor abstraction for humans, agents, and robots defined in Section 3.3 and in Equation (11)
Architectural Coherence	Responsibility separation	Layered decomposition of XR client, synchronization layer, and runtime components (Context Selector, Command Handler/Aggregate, Event Store, Projections; Figure 1). Rendering pipeline decoupled from cognitive event loop in Algorithm 2.
	Absence of cyclic dependencies	Structural analysis of event propagation graph in Section 4.1 induced by Figure 1, enforcing monotonic timestamping, role separation, and non-authoritative real-time bypass for directed acyclic event flow constraints.
	Ingress/processing/egress boundaries	Event ingress and egress points defined in Algorithm 1
	Fast vs. authoritative path separation	Verification that real-time scene updates bypass persistence while authoritative events flow through Command Handler and Event Store to Projections (Figure 1). Reality switching (Equation (5)) is affecting only render state.
Event Semantics Correctness	Event type disjointness	Formal event taxonomy defined in Section 4.3 and Table 1.
	Routing determinism	Deterministic event dispatch logic in Algorithm 1 and Equation (6).
	Sync vs. async semantics	Temporal ordering and asynchronous handling formalized via Lamport-ordered event log (10) and actor-level event decoupling (12) in the synchronization layer.
	Domain and persistence integrity	Validation of invariant enforcement in Command Handler, append-only semantics in Event Store, and deterministic Projections ensuring correct read/write separation (Figure 1). Global scene state (Equation (9)) and interaction events form a time-ordered log are formalized (Equation (10)).
Scalability & Extensibility	Agent count independence	Event-driven decoupling (12) and actor abstraction (11)
	Session scaling behavior	Coordination time complexity bounded by multi-entity decision-making time (Equation (14)). Proposition 2 states that coordination remains polynomial with bounded entity state size and fixed merge operators.
	Agent closed-loop feedback	Verification that agent outputs ($e_{ai},e_{ra}$) are correctly ingested into runtime without violating invariants or creating cycles, ensuring deterministic behavior under increasing agent numbers.

. Each construct corresponds to a formal model (algorithm, equation, or invariant) defined elsewhere in the paper, providing a structured basis for reasoning about correctness, separation of concerns, and scalability.

To provide design-time evidence that the architecture satisfies the stated requirements, a synthetic evaluation scenario (Section 5.2) symbolically executes representative multi-entity interactions. This scenario verifies that coordination, event semantics, privacy preservation, and scalability constraints are upheld under formal interaction rules.

• Coordination correctness: all events are routed, merged, and reflected in the global state $\mathbf{S}\left(t\right)$.
• Privacy compliance: raw data remains local, and federated updates enforce differential privacy.
• Scalability: decision-making and synchronization time remain within polynomial bounds, supporting increased agent counts.
• Event semantics and invariant enforcement: disjoint event types and command-handling invariants are preserved.

By formalizing global state $\mathbf{S}\left(t\right)$ and Lamport-ordered events $\mathcal{L}$, all Table 2 criteria can be reasoned about at design time. Together, the reference constructs and synthetic scenario provide design-time assurance that the architecture meets its intended requirements without requiring a full system implementation. Full empirical validation and latency–cost measurements are deferred to future work.

5.2. Synthetic Evaluation Scenario

To illustrate design-time feasibility, the minimal architecture is applied in a representative collaborative workspace comprising three entities: a human planner, an agent, and a robotic inspector. This configuration captures all core interaction classes supported by the XR input, autonomous reasoning, and sensor-driven actuation. Each entity produces interaction events, state updates, or sensor readings according to its role, formalized by Equations (15) and (16). Algorithm 4 symbolically executes this scenario by generating events per entity, enforcing causal ordering through a Lamport log, merging updates into the global synchronized state $\mathbf{S}\left(t\right)$, and propagating results through projections.

Rather than simulating concrete values, the algorithm evaluates whether architectural constraints—coordination correctness, privacy preservation, invariant enforcement, and polynomial decision-time bounds—hold under a representative multi-entity workload, thereby validating the architecture at design time.

Federated learning with differential privacy (Equation (17)) is applied to all sensor-derived data. Each agent locally computes updates to the shared DT, applies the Laplace mechanism to privatize sensitive information, and transmits only the perturbed updates to the synchronization layer. Within this synthetic scenario, the architecture can be validated for:

• Coordination correctness: All events are correctly routed, merged, and reflected in the global state $\mathbf{S}\left(t\right)$.
• Privacy compliance: Raw data never leaves local nodes; federated updates maintain differential privacy guarantees.
• Scalability and timing: Decision-making and synchronization time (Equation (14)) remain within polynomial bounds, demonstrating feasibility for larger teams.
• Event semantics and invariant enforcement: Disjoint event types (Table 1) and command-handling invariants are preserved.

Algorithm 4 Synthetic Multi-Entity Interaction Evaluation

1: procedure SyntheticEvaluation($Entities, EventRates, Projections, ϵ$)   2:     $n\leftarrow \left|Entities\right|$                  ▹ Number of active entities   3:     $\mathbf{S}\left(t\right)\leftarrow \mathrm{InitializeGlobalState}\left(\right)$   4:     $\mathcal{L}\leftarrow \mathrm{InitializeLamportLog}\left(\right)$   5:     for all $entity\in Entities$ do   6:         $r\leftarrow EventRates\left[entity\right]$   7:         $T_{\mathrm{decision}}^{\mathrm{event}}\leftarrow 0$           ▹ Initialize per-entity decision time accumulator   8:         for $i=1$ to r do   9:            $e\leftarrow \mathrm{GenerateEvent}\left(entity\right)$ 10:            $\mathcal{L}.\mathrm{Append}\left(e\right)$ 11:                                  ▹ Decision-making computation 12:            $C_{\mathrm{event}}\leftarrow O\left(n^c\right)$ 13:            $T_{\mathrm{decision}}^{\mathrm{timestep}}\leftarrow {\sum }_{entity}{T}_{\mathrm{decision}}^{\mathrm{event}}$ 14:                                       ▹ State update and merging 15:            $\mathbf{S}\left(t\right)\leftarrow \mathrm{MergeState}\left(\mathbf{S}\right(t),e)$ 16:            $\mathrm{VerifyCausality}(\mathcal{L},e)$ 17:            $\mathrm{VerifyTrust}(entity,e)$ 18:                                      ▹ Federated learning/privacy 19:            $u\leftarrow \mathrm{ComputeLocalUpdate}\left(entity\right)$ 20:            $\tilde{u}\leftarrow u+\mathrm{Lap}(\Delta f/ϵ)$ 21:            $\mathrm{TransmitUpdate}\left(\tilde{u}\right)$ 22:         end for 23:     end for 24:                          ▹ Propagation through projections/XR output 25:     for all $p\in Projections$ do 26:         $\mathrm{UpdateProjection}(p,\mathbf{S}(t\left)\right)$ 27:     end for 28:     $\mathrm{RenderXRScene}\left(\mathbf{S}\right(t\left)\right)$ 29:                                                  ▹ Scalability check 30:     $T_{\mathrm{decision}}^{\mathrm{timestep}}\leftarrow {\sum }_e{T}_{\mathrm{decision}}^{\mathrm{event}}$ 31:     if $T_{\mathrm{decision}}^{\mathrm{timestep}}>T_{\max }$ then 32:         ReportScalingLimit(n) 33:     end if 34: end procedure

6. Discussion

The layered approach separating cognition, synchronization, interoperability, and decentralization provides clear modular boundaries. This separation reduces cross-layer coupling while enabling flexible scaling, particularly when integrating AI agents or robotics. The synchronization layer acts as the critical mediator, ensuring determinism, causal message ordering, and re-playable event history.

The reality–virtuality continuum mechanism introduces a dynamic perceptual layer that can adapt to user context, device capability, or agent-driven decisions. However, enhanced perceptual flexibility raises challenges such as visual coherence across various devices, secure mixing of real and virtual visual cues, and potential manipulation risks if attackers alter rendered environments. Mitigating these requires strict integrity checks and perceptual safety audits.

A key open challenge concerns balancing user privacy with high-fidelity multi-modal sensing. Even with federated learning and differential privacy, sensor-rich XR devices generate fine-grained behavioral signatures that remain difficult to keep fully anonymous. Additional research into privacy-preserving XR compression, encrypted sensor fusion, and zero-knowledge interaction proofs is necessary.

An interesting future research direction would involve an integration of memory-centricity as an internal agent property to add persistent and continual agent memory. One such option could harness sparse representations and base-2 or base-3 systems.

6.1. Security and Privacy Challenges

Privacy-preserving mechanisms such as differential privacy are essential for protecting sensitive data in edge computing-based SC applications. Resource constraints and distributed architectures introduce additional challenges that must be considered in system design [39].

In the context of design-time evaluation, using federated learning together with the Laplace mechanism provides a synthetic yet representative validation framework: it allows us to model how updates from multiple distributed entities can be aggregated and privatized without transmitting raw sensor data, ensuring that the architecture correctly supports privacy-preserving coordination, while no real deployment is performed, this combination is sufficient to verify data flow, invariants, and event handling under privacy constraints, demonstrating feasibility within the architectural model.

Extended reality platforms supported by advanced network infrastructures introduce significant security and privacy challenges, particularly due to the handling of biometric data, immersive sensing, and distributed computation. Vulnerabilities related to virtualization, application interfaces, and data exposure must therefore be addressed as first-class design constraints in XR-enabled urban systems [40].

Differential privacy (Equation (17)) enforces a formal privacy guarantee. This approach aligns with modern recommendations for privacy-preserving sensor fusion in smart-city infrastructure [23] and for protecting biometric data collected from XR devices [32,33].

6.2. Cross-System Consistency and Scalability

Achieving deterministic state propagation across XR clients, robots, and AI agents presents complex consistency challenges. SC systems often experience contradictory data sources, asynchronous event timing, and heterogeneous device resolutions. Recent work in DT organization highlights the necessity of federated consistency layers to unify multi-platform updates [11].

INTELLECTUM resolves these challenges through: (i) Lamport-timestamped event ordering inside the synchronization layer, (ii) CRDTs for user annotations and scene edits, and (iii) chain-anchored state hashes to guarantee global consensus. This ensures replayable timelines and verifiable world-state convergence across all entities.

As the user count or sensor density increases, maintaining real-time performance becomes challenging. Large cities may produce millions of data points per minute, requiring massive parallel processing. Recent scalable digital-twin studies emphasize cloud-edge distribution as essential for maintaining responsiveness in high-load SC environments [41].

6.3. Ethical Governance

UDTs are not neutral technical artifacts but reflect underlying governance objectives, planning paradigms, and institutional priorities. Without explicitly aligning social, political, and technical dimensions, DT deployments risk internal inconsistency and limited policy relevance in urban management contexts [4]. Large-scale socio-technical systems introduce ethical risks that extend beyond technical performance, particularly when decision-making is delegated to data-driven and automated infrastructures. Responsible system design therefore requires explicit consideration of stakeholder impacts, governance structures, and long-term social impact, especially in contexts where technology mediates public life and institutional authority [42].

Algorithmic governance systems deployed in SC contexts can reinforce existing social and spatial inequalities when data asymmetries and surveillance mechanisms are insufficiently regulated. These risks highlight the need for accountability and ethical oversight in automated urban decision-making [43].

DTs increasingly function as governance instruments used by multiple urban stakeholders, including policymakers, planners, and infrastructure operators. Challenges arise when representational scope, accountability mechanisms, and institutional responsibilities are insufficiently defined [44]. Social acceptance of XR and AI interfaces in public urban environments depends on perceived usefulness, transparency, and opportunities for feedback. Participatory mechanisms that integrate citizen input into planning workflows enhance legitimacy and trust in SC deployments [45].

7. Conclusions

INTELLECTUM represents a novel metaverse architecture grounded in a reality–virtuality continuum, multi-entity coordination, and event-driven design. Beyond conceptual integration, the architecture is validated at design time through formal interaction models, explicit event semantics, and structural constraints that demonstrate feasibility prior to full system implementation.

The system advances mixed-reality platforms in three key ways. First, it enables seamless perceptual transitions across physical and virtual environments, allowing users and agents to navigate smoothly between AR, MR, and VR states through a formally defined reality-blending mechanism. Second, it supports coordinated interaction among humans, AI agents, and robotic entities via a unified actor abstraction and deterministic event routing, ensuring consistency across shared DT environments.

Design-time evaluation criteria demonstrate that these capabilities are not merely aspirational. Formal reference constructs complete the pipeline to establish architectural coherence, event semantic correctness, and scalability under increasing numbers of entities and sessions. In particular, the separation of real-time and authoritative processing paths, the absence of cyclic dependencies in the event propagation graph, and polynomial coordination bounds establish that the architecture remains tractable and deterministic without reliance on empirical tuning.

Future research directions include optimized cross-chain communication, compression techniques for reality-state transitions, and privacy-preserving mechanisms for multimodal urban data. Further work will also explore agent-based cognitive modeling and phased deployment in real-world urban environments. As a reference architecture, INTELLECTUM provides a formally grounded foundation for next-generation SCs, enabling trustworthy human–AI–robot collaboration while remaining extensible to emerging technologies and deployment scales.