Mathematical Modeling Techniques in Virtual Reality Technologies: An Integrated Review of Physical Simulation, Spatial Analysis, and Interface Implementation

Abstract

Virtual reality (VR) has emerged as a complex technological domain that demands high levels of realism and interactivity. At the core of this immersive experience lies a broad spectrum of mathematical modeling techniques. This survey explores how mathematical foundations support and enhance key VR components, including physical simulations, 3D spatial analysis, rendering pipelines, and user interactions. We review differential equations and numerical integration methods (e.g., Euler, Verlet, Runge–Kutta (RK4)) used to simulate dynamic environments, as well as geometric transformations and coordinate systems that enable seamless motion and viewpoint control. The paper also examines the mathematical underpinnings of real-time rendering processes and interaction models involving collision detection and feedback prediction. In addition, recent developments such as physics-informed neural networks, differentiable rendering, and neural scene representations are presented as emerging trends bridging classical mathematics and data-driven approaches. By organizing these elements into a coherent mathematical framework, this work aims to provide researchers and developers with a comprehensive reference for applying mathematical techniques in VR systems. The paper concludes by outlining the open challenges in balancing accuracy and performance and proposes future directions for integrating advanced mathematics into next-generation VR experiences.

1. Introduction

Virtual reality (VR) imposes unusually tight mathematical and systems constraints. Dynamics must be physically plausible. Spatial estimation must be accurate. Rendering must be photometrically consistent. Interaction must be responsive. All of these must run within millisecond budgets at 90–120 Hz to sustain presence and avoid cybersickness. Even small latency spikes or implausible physics can break immersion. VR is therefore a stringent testbed for methods spanning differential equations, optimization on manifolds, real-time rendering, and human–computer interaction.

Classical foundations remain indispensable. Rigid-body dynamics and constraint methods [1,2] govern motion and contact; illumination and projection models [3,4,5] determine image formation; quaternions and inverse kinematics (IK) on Special Euclidean Group (3D) ($\mathrm{SE}\left(3\right)$) [6,7,8,9] enable natural manipulation and embodiment. Recent advances augment rather than replace these tools: NeRFs [10] and real-time variants [11,12] learn geometry and appearance for high-quality view synthesis; differentiable rendering [13] exposes image-level gradients for joint calibration; Physics-Informed Neural Networks (PINNs) [14] embed physical laws in learning.

1.1. Scope, Perspective, and Positioning

We focus on mathematical models that enable real-time VR on consumer hardware (90–120 Hz). The survey is system-oriented. It is organized around four pillars:

(i)

Physics-based simulation: integration, constraints, collision, learned surrogates.

(ii)

Spatial representation and geometric transforms: quaternions, Lie groups, differentiable transforms, manifold optimization.

(iii)

Real-time rendering: Phong/Bidirectional Reflectance Distribution Function (BRDF), GPU pipelines, neural/differentiable rendering, perceptual strategies.

(iv)

Interaction modeling: sensing/pose, IK, hand/body tracking, constraint-based manipulation, haptics, comfort-aware control.

Authoritative surveys exist for 3D user interfaces [15], classical rendering/geometry [4,5], inverse kinematics [9], and rigid-body physics [2]. Recent surveys address neural, differentiable, and foveated rendering [10,11,12,13,16,17], but typically treat these topics in isolation or outside a VR systems perspective. Our positioning reads physics, tracking/Simultaneous Localization and Mapping (SLAM), rendering, and interaction together, framed by VR-specific constraints (quality–latency–memory trade-offs, motion-to-photon, reprojection error, SSQ) and by cross-module interfaces (e.g., $\mathrm{SE}\left(3\right)$ tracking feeding differentiable rendering and avatar controllers).

1.2. Contributions and Methodology

This article is a survey; contributions are curatorial and integrative:

(i)

We organize stability-oriented practices for VR physics. We clarify when semi-implicit/symplectic integrators are preferable. We explain how constraint projection maintains plausibility under contact. We identify where perceptually gated adaptivity can safely reduce cost. We also situate differentiable simulators within this toolbox.

(ii)

We connect literatures on manifold optimization, $\mathrm{SE}\left(3\right)$ SLAM, differentiable image formation, and neural scene fields. We present an end-to-end view for tracking, calibration, and reconstruction. We include a taxonomy and typical failure modes for avatars and hand–object interaction.

(iii)

We review hybrid and neural rendering for dual-eye 90/120 Hz VR. We distill rules of thumb for the quality–latency–memory trade-off (e.g., radiance caching, network factorization, perceptual scheduling).

(iv)

We align methods in physics, rendering, and interaction (IK, haptics, personalized avatars) with user-facing and system metrics. We link algorithm families to time-to-grasp, slip rate, embodiment, Absolute Trajectory Error (ATE)/Relative Pose Error (RPE), reprojection error, and motion-to-photon latency.

(v)

We compile checklists and a consolidated benchmark table (Appendix A). We outline a near-term (1–2 year) agenda with concrete goals. Targets include numerical energy drift $\le 1\%$ on standard scenes. We seek average reprojection error $<1$ px. We target end-to-end motion-to-photon $<20$ ms. We aim for 20–$30\%$ lower SSQ under standardized protocols. To ground the discussion in practical device constraints, Appendix A.1 reports a headset specification summary (

Table A1. VR headset comparison (manufacturer specifications). Device weight, compute, and display specifications used as practical constraints for VR system design. Data compiled from manufacturer documentation and independent benchmarks (2024–2025).

Device	Weight (g)	Chipset	RAM	Display	Resolution (per Eye)	Refresh (Hz)
Meta Quest 3	515	Snapdragon XR2 Gen2	8 GB	Fast-switch LCD	$2064\times 2208$	72–120
Apple Vision Pro	750–800	Apple M5/R1	16 GB	micro-OLED	$3660\times 3200$	90–120
PlayStation VR2	560	Host-dependent	–	OLED	$2000\times 2040$	120

).

To provide a high-level roadmap, Figure 1 summarizes the simplified VR system pipeline considered in this survey (hardware → tracking → physics/IK → rendering → display) and maps each block to the corresponding sections and representative equations. In particular, we later illustrate the numerical stability of the physics update in Figure 2 and detail the rendering pipeline in Figure 3.

Selection criteria. We surveyed 2020–2025 work in SIGGRAPH, TOG, Eurographics, IEEE VR, TVCG, Virtual Reality, CHI, CVPR, ICCV, ECCV, NeurIPS, and ICRA, prioritizing: (i) real-time constraints (≥90 Hz or ≤20 ms), (ii) explicit mathematical formulations, (iii) quantitative validation, and (iv) integration/deployment considerations. Foundational references are included where they clarify principles or baselines.

1.3. Reader’s Guide and Conventions

Reader’s guide. Section 2 covers numerical integration, constraint satisfaction, collision detection/response, and stability–performance trade-offs. Section 3 treats coordinate frames, rotation representations, differentiable transforms, manifold optimization, and pose-graph estimation. Section 4 spans classical to neural/differentiable rendering and perceptual strategies. Section 5 reviews sensing, IK, haptics, and comfort-aware control. Section 6 sets near-term targets; Section 7 synthesizes implications.

Notation and conventions. Bold letters denote vectors/matrices; $\mathrm{SO}\left(3\right)$/$\mathrm{SE}\left(3\right)$ denote rotations/rigid motions; unit quaternions represent orientation. We use standard reprojection/trajectory metrics (ATE/RPE), SSQ for comfort, and per-eye throughput (resolution × refresh) for system reporting.

2. Physics-Based Simulation in VR

VR immersion depends not only on visual fidelity but also on physically consistent responses under tight frame budgets. This section formalizes equations of motion and constraints, compares time integration schemes under accuracy-stability-structure trade-offs, summarizes contact/friction as non-smooth dynamics, and details perceptual/system-aware adaptivity and learning-augmented simulators relevant to real-time deployment.

Table 1. Domain-specific modeling priorities in VR applications. Ratings (1–5) indicate relative emphasis under typical requirements; key constraints drive system-level trade-offs discussed in later sections.

Domain	Physics	Tracking	Rendering	Key Constraint
Medical	5	4	3	Haptic fidelity (kHz)
Gaming	3	4	5	90 Hz minimum
Industrial	3	5	4	Sub-cm accuracy
Social VR	2	5	3	Network latency

provides an at-a-glance summary of modeling priorities across representative VR application domains. The ratings indicate relative emphasis (1–5) in terms of typical system requirements rather than a universal ranking. These priorities guide the focus of the subsequent sections on tracking, physics/IK, and rendering, where we detail the corresponding mathematical formulations.

2.1. VR Strengths and Limitations (at a Glance)

• Strengths

  –

  Perceptual realism: physically consistent responses increase presence and make user action outcomes predictable.

  –

  Interaction stability: constraint-aware solvers reduce jitter/failure during grasp/manipulation [2].

  –

  Composability: shared rigid/soft formulations unify haptics, avatars, and environment logic.

• Limitations

  –

  Strict frame budgets: solver work must fit $\approx 11.1$ ms @ 90 Hz (8.3 ms @ 120 Hz); physics LOD/scheduling are mandatory [18,19].

  –

  Stability–accuracy trade-off: Semi-implicit/projection schemes preserve stability but bias trajectories. Higher-order non-symplectic methods are costlier [2,20].

  –

  Contact/constraint overhead: non-smooth friction/contact induces LCP/SOCP solves; under-resolved contacts cause visible interpenetration/haptic artifacts [21,22].

  –

  Latency coupling: physics–tracking–render clocks interact; motion-to-photon and prediction errors degrade comfort [5].

2.2. Equations of Motion and Constraints

This subsection states the equations of motion for a rigid multibody system and how equality/inequality constraints enter the model. We fix notation for the state $(q,v)$, separate model terms $M, C, f_{\mathrm{ext}}$, and introduce Lagrange multipliers for constraints; these definitions will be used throughout the section.

Classical physically based modeling formulations established the rigid-body dynamics and constraint foundations that still underpin real-time physics engines used in modern VR [23].

Let generalized coordinates $q\in {\mathbb{R}}^n$ and velocities $v=\dot{q}$. In the absence of constraints, the equations of motion read

$$M\left(q\right)\dot{v}+C(q,v)=f_{\mathrm{ext}}(q,v,t),$$

(1)

with mass matrix M, Coriolis/gyroscopic term C, and external forces $f_{\mathrm{ext}}$.

Intuitively, Equation (1) is the n-DOF generalization of Newton’s law: $M\left(q\right)$ maps accelerations to forces, while $C(q,v)$ gathers velocity-dependent inertial effects that become prominent in coupled or fast motions.

Holonomic constraints $g\left(q\right)=0$ (e.g., joints) and unilateral nonpenetration $g_n\left(q\right)\ge 0$ (contacts) impose algebraic conditions. Introducing Lagrange multipliers $\lambda$ for the constraints yields the constrained system

$$\begin{array}{cc}\hfill M\left(q\right)\dot{v}+C(q,v)& =f_{\mathrm{ext}}(q,v,t)+J{\left(q\right)}^{\top }\lambda ,\hfill \\ \hfill g\left(q\right)& =0,J\left(q\right)=\partial g/\partial q,\hfill \end{array}$$

(2)

where J is the constraint Jacobian. Constraint drift can be reduced by Baumgarte stabilization (enforcing $\dot{g}+\alpha g=0$ in the integrator) or by post-stabilization, i.e., projecting $(q,v)$ back to the constraint manifold after each step [1,2].

A common post-stabilization step applies a minimal correction that projects the tentative configuration $\tilde{q}$ back onto the constraint manifold:

$$q\leftarrow \tilde{q}+\Delta q,\Delta q=-M{\left(\tilde{q}\right)}^{-1}J{\left(\tilde{q}\right)}^{\top }{\left(J\left(\tilde{q}\right)M{\left(\tilde{q}\right)}^{-1}J{\left(\tilde{q}\right)}^{\top }\right)}^{-1}g\left(\tilde{q}\right),$$

(3)

where $J\left(q\right)=\partial g/\partial q$.

2.3. Time Integration: Accuracy, Stability, and Structure

This subsection contrasts commonly used time-stepping schemes for interactive VR physics and graphics. We focus on how order of accuracy, stability, and structure preservation trade off under real-time budgets. Let $\Delta t$ be the fixed timestep and let $a(q,v)$ denote the instantaneous acceleration implied by forces and constraints.

Explicit Euler. The explicit Euler integration is formulated as follows:

$$v_{k+1}=v_k+\Delta ta(q_k,v_k),q_{k+1}=q_k+\Delta t{v}_k.$$

(4)

It is first-order accurate and cheap, but it tends to drift in energy over long horizons.

Symplectic (semi-implicit) Euler. The symplectic Euler integration is formulated as:

$$v_{k+1}=v_k+\Delta ta(q_k,v_k),q_{k+1}=q_k+\Delta t{v}_{k+1}.$$

(5)

This scheme preserves a discrete symplectic structure and often exhibits bounded long-horizon energy error for Hamiltonian systems. To illustrate the long-horizon stability of Equation (4), Figure 2 compares the relative energy error of a simple pendulum using explicit Euler, symplectic (semi-implicit) Euler (Equation (4)), and RK4 under the same step size $\Delta t$.

Position Verlet. This simple equation gives the position Verlet method:

$$q_{k+1}=2q_k-q_{k-1}+\Delta t^{2}a\left(q_k\right).$$

(6)

It is second-order accurate and symplectic for conservative forces and is widely used for cloth and soft-constraint models.

High-order non-symplectic (RK4). Classical RK4 is fourth-order accurate per step but non-symplectic and costlier; it suits small, nonstiff subsystems (e.g., camera oscillators) when local accuracy outweighs structure preservation [20].

In practice, semi-implicit updates are often coupled with constraint projection methods such as position-based dynamics (PBD) and extended position-based dynamics (XPBD) to maintain stability under contact bursts [2,24,25].

2.4. Contact, Friction, and Non-Smooth Dynamics

This subsection summarizes the discrete-time treatment of rigid contact and Coulomb friction used in real-time VR simulators. We adopt an impulse and velocity level view, which handles intermittent contact, sticking and sliding, and naturally leads to complementarity or cone projection problems.

Normal contact. Let $g_n$ denote the end-of-step normal gap (non-penetration requires $g_n\ge 0$) and let ${\lambda }_n\ge 0$ be the normal impulse delivered over the step. Normal contact is enforced by the complementarity condition

$$0\le {\lambda }_n\perp g_n\ge 0,$$

(7)

meaning either a positive gap with zero impulse, or zero gap with a compressive impulse [1,21].

Tangential friction. With coefficient $\mu$, the Coulomb cone is

$$\parallel {\lambda }_t\parallel \le \mu {\lambda }_n,$$

(8)

with sticking if the post-impact tangential velocity $v_t=0$ lies strictly inside the cone, and sliding on the boundary with $v_t$ opposite the friction direction (maximum dissipation).

Discrete formulations and solvers. Approximating the circular cone by a friction pyramid yields a linear complementarity problem (LCP); keeping the cone gives a second-order cone program (SOCP). Common real-time solvers include projected Gauss–Seidel and cone projection (PGS/CP), proximal or alternating direction method of multipliers (ADMM) variants, and interior-point methods

Proximity queries. Accurate contact generation requires robust distance queries and feature pairs. For rigid models, Gilbert–Johnson–Keerthi (GJK) algorithm with Expanding Polytope Algorithm (EPA) refinement and bounding volume hierarchies (BVHs) are standard; for deformables, signed distance fields or volumetric proxies are widely used [26,27,28].

2.5. Perceptual- and System-Aware Adaptivity

This subsection formalizes adaptivity for real-time VR as four ingredients: error-controlled timestepping, contact-aware substepping, gaze-driven physics LOD, and frame-budget scheduling. The goal is to meet a per-frame budget of B (e.g., $B\approx 11.1$ ms at 90 Hz) without perceptible artifacts [2,18,19,22].

(i)

Error-controlled timestepping.

Let the integrator have local order p, and let ${\widehat{e}}_k$ be an estimate of the local truncation error at step k (from an embedded pair or a residual). For a user or device tolerance ${\tau }_{\mathrm{phys}}$,

$${\widehat{e}}_k\le {\tau }_{\mathrm{phys}}\Rightarrow \Delta t_{k+1}=\Delta t_k\mathrm{safety}{\left(\frac{{\tau }_{\mathrm{phys}}}{{\widehat{e}}_k}\right)}^{{\displaystyle \frac{1}{p+1}}},$$

(9)

with $\mathrm{safety}\in (0,1)$ to prevent step rejection cascades. This keeps integration error below a perceptual threshold while exploiting slack when motion is slow.

(ii)

Contact-aware substepping.

For a candidate contact with normal gap $g_n>0$ and closing speed $v_n^{-}\ge 0$, choose the number of substeps m inside the frame step $\Delta t$ so that predicted closure per substep is a fraction $\eta \in (0,1)$ of the gap:

$$m\ge ⌈{\displaystyle \frac{v_n^{-}\Delta t}{\eta g_n+\epsilon }}⌉,\Delta t_{\mathrm{sub}}=\Delta t/m,$$

(10)

with small $\epsilon$ to avoid division by zero. This reduces penetrations without globally shrinking $\Delta t$ during brief contact bursts.

(iii)

Gaze-driven physics LOD.

Let $\pi :{\mathbb{R}}^3\to {\mathbb{R}}^2$ be the rendering projection and $J_i=\partial \pi /\partial x$ evaluated at object i. To bound screen-space motion by an eccentricity-dependent tolerance ${\tau }_{\mathrm{pix}}\left({\theta }_i\right)$ (from eye-tracking),

$$\parallel J_i\delta x_i\parallel \le {\tau }_{\mathrm{pix}}\left({\theta }_i\right),\delta x_i\approx v_i\Delta t_i,$$

(11)

which yields a per-object update interval

$$\Delta t_i\le {\displaystyle \frac{{\tau }_{\mathrm{pix}}\left({\theta }_i\right)}{\parallel J_i{v}_i\parallel }}.$$

(12)

Equivalently, define a perceptual weight $w_i=w\left({\theta }_i\right)$ (e.g., $w\left(\theta \right)=exp(-\beta \theta )$) and set per-object iteration counts $I_i\propto w_i$ or update rates $r_i=1/\Delta t_i$ that decrease with eccentricity [19].

(iv)

Frame-budget scheduling.

Let $\mathcal{T}$ be tasks in the current frame (constraint batches, cloth solver iterations, background rigid updates, etc.). Each task $t\in \mathcal{T}$ has cost $c_t$ and perceptual utility $u_t$ (higher near fovea, for large screen-space motion, or for audible haptic coupling).

We allocate work by

$$\underset{\left\{x_t\right\}}{max}\sum _{t\in \mathcal{T}}{u}_t{x}_t\mathrm{subject}\mathrm{to}\sum _{t\in \mathcal{T}}{c}_t{x}_t\le B,x_t\in \mathbb{N},$$

(13)

a knapsack-style problem. A greedy policy using ratio $u_t/c_t$ is effective in real time; engine backends (e.g., DOTS, Chaos, Flex) realize this by prioritizing foreground constraints and deferring low-$u_t$ tasks to worker queues [2].

Clock Alignment

Let $T_{\mathrm{sim}}+T_{\mathrm{ren}}+T_{\mathrm{queue}}\le B$. We reduce perceived latency by predicting the presented state to the next vsync time $t_v$:

$$\widehat{x}\left(t_v\right)=x_k+v_k(t_v-t_k),$$

(14)

and by choosing $\Delta t$ via (9) so that jitter in $T_{\mathrm{sim}}$ does not spill over B.

2.6. Learning- and Gradient-Based Simulation

Differentiable physics layers embed simulation inside gradient-based pipelines for system identification, control, and inverse problems; Recent differentiable solvers that explicitly handle contact (and, when available, frictional effects) improve gradient quality for learning-based control and inverse problems [29]. Such advances make physically grounded optimization more practical within real-time VR pipelines.

Moreover, contact is often regularized (compliance/soft constraints) to ensure stable gradients [30,31]. Physics-Informed Neural Networks incorporate PDE residuals into the loss,

$${\mathcal{L}}_{\mathrm{PINN}}={\mathcal{L}}_{\mathrm{data}}+\lambda {\parallel \mathcal{N}(q,v,\dot{v})-f\parallel }^2,$$

(15)

serving as surrogates for expensive substeps (e.g., local solves in reduced FEM) or for environment response fields in digital twins [14,32,33]. These hybrids can reduce substep counts while meeting comfort constraints when validated against baselines.

Deployment in real-time VR must additionally account for tail latency and out-of-distribution failures; practical guardrails and fallback strategies are discussed in Section 6.8.

2.7. Practical Guidance and Trade-Offs

Based on

Table 2. Integration schemes under real-time VR constraints. The typical use cases reflect common VR workloads, ranging from low-priority background props to articulated characters and constraint-rich soft objects. The local error and energy behavior indicate long-horizon stability (e.g., drift versus bounded oscillations), while per-step cost determines achievable update rates under stereo frame budgets. See Section 2.7 for stability-oriented discussion and practical selection guidelines.

Method	Local Error	Energy Behavior	Per-Step Cost	Typical Use in VR
Explicit Euler	$\mathcal{O}(\Delta t^2)$ (1st order global)	Drift (non-symplectic)	Very low	Background props, cheap particles; avoid long horizons
Symplectic Euler	$\mathcal{O}(\Delta t)$	Bounded drift (symplectic)	Very low	Rigid stacks, ragdolls with constraint projection [2]
Verlet (position)	$\mathcal{O}(\Delta t^3)$ (2nd global)	Good for conservative forces (symplectic)	Low	Cloth/strings, soft constraints with XPBD-style projection [24,25]
RK4	$\mathcal{O}(\Delta t^5)$ (4th global)	Non-symplectic	Medium–high	Small ODEs requiring accuracy (camera oscillators, effects) [20]

and the preceding discussion, several practical guidelines emerge for real-time VR: structure-preserving (symplectic) schemes are generally preferable for long-horizon stability; adaptive substepping near impacts mitigates penetrations without shrinking the global step; per-frame constraint projection helps maintain feasibility under contacts; and perceptual gating reduces workload by lowering update rates for off-gaze or low-saliency bodies [18,19].

To ground the above dual-rate scheduling principles in a concrete VR medical scenario, we summarize a representative deployment in Box 1 (see Appendix A.2 for full details).

Box 1. Illustrative Deployment: Medical VR Surgical Simulator

Context: A neurosurgery training system requires haptic feedback at 1 kHz while rendering at 90 Hz on Quest 3 (Table A1).

Mathematical solution:

• Physics (Section 2): Reduced-order FEM with 50 modal basis functions (Equation (1), $\mathbf{M}\ddot{\mathbf{q}}+\mathbf{C}={\mathbf{f}}_{\mathrm{ext}}$). Modal reduction: 10K tetrahedral elements → 50 modes, achieving 0.8 ms update at 90 Hz.
• Haptics (Section 5.4): Passivity controller (Equations (66) and (67)) with adaptive damping maintains $E\left[k\right]\ge 0$ over 5-min procedures, preventing instability under variable rendering latency.
• Rendering (Section 4): Per-vertex shader skinning from modal weights; budget allocation: 3 ms physics + 6 ms rendering + 2 ms margin $=11$ ms at 90 Hz.

Measured outcomes ($n=12$ surgeons, 20 procedures):

Metric	Target	Achieved	Ref.
Haptic rate	1 kHz	950–1000 Hz	Section 5.4
Visual rate (P95)	90 Hz	88–90 Hz	Section 6
Force fidelity	<10% error	8.1% RMSE	Equation (68)

Key lesson: Modal reduction (Section 2.6) is essential—full FEM cannot meet dual refresh rates. Decoupled visual/haptic threads (Section 5.4) prevent mutual interference. See Appendix A.2 for full deployment details including design rationale and measured performance breakdown.

2.8. Summary of Section 2

Table 3. Section 2 at-a-glance: core physics formulations, their practical implications for real-time VR, and representative references. Each row summarizes the mathematical form, typical stability/performance trade-offs, and the system-level consequences that matter at VR update rates (e.g., feasibility under constraints and long-horizon energy behavior). Cross-references point to detailed derivations and implementation notes.

Topic	Core Formulation	Practical Implications	Canonical Refs.
Dynamics & constraints	$M\dot{v}+C=f_{\mathrm{ext}}+J^{\top }\lambda$; algebraic constraints $g\left(q\right)=0$, $g_n\left(q\right)\ge 0$; stabilization via Baumgarte or post-step projection.	Fixes state/force notation used throughout. Projection keeps feasibility and reduces drift under real-time steps.	[1,2]
Time integration	Semi-implicit/symplectic, Verlet, RK4 (accuracy vs. structure).	Use symplectic + projection for long-horizon stability; reserve RK4 for small, non-stiff subsystems when accuracy dominates.	[20,24,25]
Contact & friction	Complementarity $0\le {\lambda }_n\perp g_n\ge 0$; Coulomb cone $\parallel {\lambda }_t\parallel \le \mu {\lambda }_n$; LCP/SOCP formulations; collision via GJK/EPA, BVHs; deformables via SDF/volumetric proxies.	Choose PGS/CP for speed or ADMM/interior-point for tighter solves on small sets. Robust proximity queries are critical for stable contacts.	[21,26,27,28]
Perceptual/system adaptivity	Saliency/occlusion/gaze-gated LOD and update-rate scheduling to meet frame budget B (e.g., $B\approx 11.1$ ms @ 90 Hz).	Prioritize foreground constraints; throttle off-gaze/low-utility tasks to cut misses/jitter without visible artifacts.	[2,18,19,22]
Integrator trade-offs	See Table 2.	Compare accuracy, stability, and cost of representative schemes under VR budgets.	—

summarizes the core formulations, practical implications, and canonical references introduced in Section 2 under real-time VR constraints.

3. Spatial Representation and Geometric Transformations in VR

Classical spatial modeling—coordinate frame definitions, transform chains, and rotation operators—remains the backbone of VR systems. What has changed since 2020 is how these primitives are organized, differentiated, and learned in modern pipelines. This section reviews the minimal classical tools needed for VR, then focuses on recent trends that make these tools robust and learnable at scale.

3.1. Coordinate Frames and Transform Chains

This subsection recalls the minimal coordinate-frame conventions and transform chains used throughout VR pipelines. We focus on composing and inverting rigid transforms for head, hand, and world frames, and on bookkeeping that avoids drift in long chains.

Scene graphs in VR arrange objects by hierarchical frames; correctness depends on composing translation, rotation, and scale with numerically stable conventions. Practical recipes for transform order, handedness, and matrix conditioning are well-documented in graphics texts and VR geometry primers [5,34]. In practice, systems favor right-handed world frames with explicit documentation of pre-/post-multiplication and column-/row-major storage to avoid ambiguity during engine integration [34].

3.2. Rotations: From Euler Angles to Quaternions and Lie Groups

We review rotation representations with an eye to interactive stability: Euler angles are compact but suffer from singularities, while quaternions and Lie-group updates provide smooth composition and interpolation for head, hand, and camera control. We outline when each is preferable in real-time systems and prepare the notation used later.

Euler angles are compact but suffer from gimbal singularities; quaternions support smooth interpolation (SLERP) and numerically robust composition for head–pose, hand–pose, and camera control [6,7]. Surveys on 3D orientation (attitude) representations compare rotation matrices, quaternion/Euler-parameter forms, and exponential maps and frame rotations as elements of the Lie group $\mathrm{SO}\left(3\right)$ with minimal updates in its tangent space [35].

This Lie-group view is advantageous for VR tracking and control because it enforces orthogonality and unit norm by construction, avoids parameter singularities and redundancy, yields consistent Jacobians for EKF/SLAM/ICP, and reduces long–chain drift via manifold retraction [5].

Consequently, modern systems favor quaternion updates or Lie–algebra increments ($R\leftarrow Rexp({\left[\omega \right]}_{\times }\Delta t)$ ) within $\mathrm{SO}\left(3\right)$ and $\mathrm{SE}\left(3\right)$ to maintain stability at interactive rates [5,6].

3.3. Differentiable Transforms and Inverse Graphics

Modern VR stacks expose camera and object transforms to gradient-based optimization. This subsection formalizes differentiable parameterizations for extrinsics/intrinsics and shows how they plug into photometric objectives without breaking geometric constraints.

A major shift is differentiable transforms that permit gradients to flow through rendering and geometry. Early differentiable rendering frameworks established approximate yet useful Jacobians for camera and lighting parameters [13]. Neural implicit scene models (e.g., NeRF and its fast variants) embed pose and calibration variables into optimization, enabling view synthesis and on-the-fly relighting for VR telepresence and scene capture [10,12].

These pipelines rely on stable pose/warp parameterizations (quaternions, axis–angle) so that gradient-based solvers remain well-conditioned during joint optimization of geometry and appearance [10,13].

3.4. Learning on Manifolds and Non-Euclidean Domains

Many geometry spaces in VR are non-Euclidean (poses on SO(3)/SE(3), shapes on low-dimensional manifolds). We summarize practical manifold-aware tools used for tracking, retargeting, and avatar personalization, emphasizing what matters at interactive rates.

Geometric deep learning extends classical signal processing to meshes, graphs, and point clouds—data domains common in VR avatars, environments, and props. Convolutional operators on manifolds support curvature-aware feature extraction for registration and morphable modeling [36]; recent surveys generalize these ideas to groups, graphs, and gauges with principled handling of symmetry and coordinate choices [37]. In human-centric VR, kinematic consistency is enforced by putting learning inside a model-based loop (e.g., SMPL/SMPL-X fitting), improving 3D pose/shape reconstruction from monocular or sparse sensors [38,39].

3.5. Pose Graphs, SLAM, and Online Estimation for VR

We introduce pose-graph optimization for keeping content world-locked: nodes are 6-DoF poses and edges are relative measurements with covariances. We then give the standard SE(3) objective and explain how these estimators stabilize mixed-reality overlays in real time.

Interactive VR scenes benefit from real-time pose-graph optimization and differentiable tracking to keep virtual content world-locked (anchored in a fixed world frame so objects do not move with the user’s head), rather than head-locked (attached to the display frame). A pose graph is a sparse graph whose nodes are 6-DoF poses $T\in \mathrm{SE}\left(3\right)$ and whose edges encode relative-pose measurements with covariances.

A standard objective uses Lie-group residuals via the logarithm map:

$$\underset{\{T_i\in \mathrm{SE}\left(3\right)\}}{min}\sum _{(i,j)\in \mathcal{E}}\rho \left({∥{\Sigma }_{ij}^{-1/2}log\left(Z_{ij}^{-1}{T}_i^{-1}{T}_j\right)∥}_2^2\right),$$

(16)

Equation (15) minimizes weighted relative-pose residuals over edges $(i,j)\in \mathcal{E}$ by mapping the group discrepancy $Z_{ij}^{-1}{T}_i^{-1}{T}_j$ to a 6D tangent-space error via $log(·)$, and scaling it by the uncertainty ${\Sigma }_{ij}^{-1/2}$ while robustifying with $\rho (·)$. In VR, this is used to suppress drift and keep anchors world-locked by enforcing global consistency across many local motion constraints (e.g., loop closures and multi-sensor factors) in Section 3.5.

Recent deep SLAM systems couple learned features with geometric bundle adjustment and robust $\mathrm{SE}\left(3\right)$ updates, yielding accurate trajectories under fast motion and low texture [40]. These estimators integrate with differentiable rendering and neural scene fields (Section 3 and Section 4), enabling joint refinement of poses, intrinsics/extrinsics, scene fields, and rendering parameters at runtime.

3.6. Summary of Section 3

Table 4. Section 3 at-a-glance: spatial representations, differentiable image formation, and manifold-aware estimation relevant to VR tracking and calibration. The summary emphasizes structure-preserving pose updates and differentiable objectives that enable online refinement under noisy sensing. These elements support world-locked content, stable anchors, and reproducible evaluation in Section 6.

Topic	Core Idea/Formulation	Practical Implication for VR	Canonical Refs
Transforms & frames	On-manifold pose updates ($R\leftarrow Rexp\left({\left[\delta \varphi \right]}_{\times }\right)$), unit-quaternion orientation with normalization; stable $\mathrm{SE}\left(3\right)$ chaining.	Reduces drift and gimbal issues at 90/120 Hz; improves numerical stability in long transform chains and resampling.	[6,34,35]
Differentiable image formation & neural fields	Photometric losses with Jacobians w.r.t. $\mathrm{SE}\left(3\right)$ pose/intrinsics; NeRF-style radiance fields for view synthesis and calibration.	Enables gradient-based tracking/relocalization and online calibration; ties rendering directly to estimation quality.	[10,12,13]
Manifold-aware learning (meshes, avatars)	Geometric DL on surfaces/graphs (e.g., Laplace–Beltrami, spectral/mesh operators); differentiable body/face models.	Respects non-Euclidean structure; better generalization and lower retargeting error for hand-object and avatar tasks.	[36,37,38,39]
SLAM & pose-graph optimization	On-manifold bundle adjustment and dense–sparse fusion for robust pose graphs under motion.	Tighter online estimates for world-locked content; reduced drift for stable anchoring in interactive scenes.	[40]

summarizes the main spatial representations and differentiable image-formation tools from Section 3, emphasizing their practical impact on VR tracking, calibration, and avatar control.

4. Real-Time Rendering and Mathematical Models in VR

Rendering is the terminal stage of the VR pipeline that translates a mathematically defined scene into the visual stimuli perceived by the user. Unlike offline graphics, VR imposes strict constraints on latency, frame rate, and stability, since delays or visual artifacts can degrade presence or induce discomfort. This section organizes the mathematics behind real-time rendering and highlights how recent developments extend classical models [4,5].

4.1. Classical Illumination Models and Shading

This subsection summarizes core shading models and shows how the rendering integral is evaluated efficiently on the GPU.

4.1.1. Phong Shading

The Phong model decomposes shading into ambient, diffuse, and specular terms [3]:

$$I=k_a{I}_a+k_d(L·N)I_{\ell }+k_s{(R·V)}^n{I}_{\ell },$$

(17)

where $k_a,k_d,k_s$ are material coefficients, L is the light direction, N the surface normal, R the reflection of L about N, and V the view direction.

4.1.2. Rendering Equation (Physically Based Rendering)

Physically based rendering (PBR) adopts energy-conserving microfacet BRDFs [41]:

$$L_o(p,{\omega }_o)={\int }_{\mathrm{\Omega }\left(n\right)}{f}_r(p,{\omega }_i,{\omega }_o)L_i(p,{\omega }_i)max(0,n·{\omega }_i)d{\omega }_i.$$

(18)

Here $L_o$ is the outgoing radiance toward ${\omega }_o$ at point p with normal n, $L_i$ is incident radiance from ${\omega }_i$ over the hemisphere $\mathrm{\Omega }\left(n\right)$, $f_r$ is the BRDF, and $max(0,n·{\omega }_i)$ is the cosine term.

4.1.3. Monte Carlo Evaluation of the Hemisphere Integral

On the GPU, the integral is approximated by Monte Carlo with importance sampling [42]:

$${\widehat{L}}_o={\displaystyle \frac{1}{N}}\sum _{k=1}^N{\displaystyle \frac{f_r(p,{\omega }_k,{\omega }_o)L_i(p,{\omega }_k)max(0,n·{\omega }_k)}{p\left({\omega }_k\right)}},$$

(19)

where samples ${\omega }_k\sim p\left(\omega \right)$ are drawn from a proposal p matched to the integrand. Practical choices: cosine-weighted sampling for diffuse and sampling the microfacet normal distribution (e.g., GGX (Ground Glass × unknown)/Trowbridge–Reitz) for glossy lobes [43]. For scenes with both emitters and glossy BRDFs, multiple importance sampling blends light- and BRDF-based proposals to reduce variance [42].

4.1.4. Real-Time Approximations in Engines

Typical pipelines combine (i) next-event estimation for direct lighting, (ii) a few BRDF-importance samples for specular response, and (iii) denoising or temporal accumulation for residual noise. Preintegrated approximations (e.g., split-sum environment BRDF via small LUTs for geometry and Fresnel terms) amortize cost to near constant time in PBR engines [4,5].

4.2. Camera Models and Projection Matrices

This subsection formalizes the pinhole and off-axis (asymmetric) stereo cameras used in VR, derives per-eye projection from headset geometry, and gives the math for lens predistortion and timewarp.

4.2.1. Symmetric Perspective

With intrinsics $f_x,f_y$, and principal point $(c_x,c_y)$,

$$K=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right],P_{\mathrm{sym}}=\left[\begin{array}{cccc}{\displaystyle \frac{2n}{w}}& 0& 0& 0\\ 0& {\displaystyle \frac{2n}{h}}& 0& 0\\ 0& 0& {\displaystyle \frac{f+n}{n-f}}& {\displaystyle \frac{2fn}{n-f}}\\ 0& 0& -1& 0\end{array}\right].$$

(20)

where $n,f$ are near/far distances and $w,h$ are the near-plane width/height.

4.2.2. Off-Axis (Asymmetric) per-Eye Frustum

With near-plane bounds $l,r,b,t$ in camera space,

$$P_{\mathrm{off}}=\left[\begin{array}{cccc}{\displaystyle \frac{2n}{r-l}}& 0& {\displaystyle \frac{r+l}{r-l}}& 0\\ 0& {\displaystyle \frac{2n}{t-b}}& {\displaystyle \frac{t+b}{t-b}}& 0\\ 0& 0& {\displaystyle \frac{f+n}{n-f}}& {\displaystyle \frac{2fn}{n-f}}\\ 0& 0& -1& 0\end{array}\right].$$

(21)

Place the screen/lens image plane at distance d with physical rectangle $[L,R]\times [B,T]$ (meters). Then

$$l=n{\displaystyle \frac{L}{d}},r=n{\displaystyle \frac{R}{d}},b=n{\displaystyle \frac{B}{d}},t=n{\displaystyle \frac{T}{d}}.$$

(22)

If the per-eye horizontal optical center shift is $s_e$ (IPD and lens center), with panel half-width $W/2$, then

$$L=-{\displaystyle \frac{W}{2}}-s_e,R={\displaystyle \frac{W}{2}}-s_e,s_e=\pm \frac{\mathrm{IPD}}{2}-c_x^{\left(\mathrm{lens}\right)},$$

(23)

where the sign + is for the right eye and − for the left eye.

4.2.3. Lens Distortion (Predistortion for VR)

Let $(x,y)$ be normalized image-plane coordinates and $r^2=x^2+y^2$. The Brown–Conrady model writes [44,45]

$$\begin{array}{cc}\hfill x_d& =x\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)+2p_{1}xy+p_2\left(r^2+2x^2\right),\hfill \\ \hfill y_d& =y\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)+p_1\left(r^2+2y^2\right)+2p_{2}xy.\hfill \end{array}$$

(24)

VR runtimes render to a predistorted image using $(x,y)\mapsto (x_d,y_d)$ (or its inverse via a LUT) so that post-lens appearance is undistorted [5].

4.2.4. Timewarp (Orientation-Only and Depth-Aware)

Let K be intrinsics, $R_{\mathrm{pred}}$ the head rotation used for rendering, and $R_{\mathrm{now}}$ the most recent rotation. Orientation-only timewarp:

$$\tilde{\mathbf{u}}\propto K\left(R_{\mathrm{now}}{R}_{\mathrm{pred}}^{-1}\right)K^{-1}\mathbf{u},$$

(25)

for homogeneous pixel $\mathbf{u}={(u,v,1)}^{\top }$; $\tilde{\mathbf{u}}$ samples the rendered color buffer. With per-pixel depth $D\left(\mathbf{u}\right)$, reconstruct $X=D{K}^{-1}\mathbf{u}$, apply $\Delta T=T_{\mathrm{now}}{T}_{\mathrm{pred}}^{-1}$, and reproject:

$${\mathbf{u}}^{\prime }\propto K\left(\Delta TX\right).$$

(26)

Both warps reduce apparent motion-to-photon latency; use the same per-eye $P_{\mathrm{off}}$ in the original render and the warp.

4.3. GPU Pipelines and Shader Mathematics

This subsection isolates shader-side math that most directly affects VR image quality: perspective-correct interpolation, depth mapping/precision, normal-space transforms, and texture filtering with MIP selection.

4.3.1. Perspective-Correct Interpolation

Let a triangle have screen-space barycentrics $\alpha ,\beta ,\gamma$ with $\alpha +\beta +\gamma =1$ and clip-space $w_i$ at vertices $i\in \{1,2,3\}$. For an attribute a,

$$a_{\mathrm{persp}}={\displaystyle \frac{\alpha {\displaystyle \frac{a_1}{w_1}}+\beta {\displaystyle \frac{a_2}{w_2}}+\gamma {\displaystyle \frac{a_3}{w_3}}}{\alpha {\displaystyle \frac{1}{w_1}}+\beta {\displaystyle \frac{1}{w_2}}+\gamma {\displaystyle \frac{1}{w_3}}}},$$

(27)

which avoids the distortion of affine (screen-linear) interpolation [4,46].

4.3.2. Depth Mapping and Precision

With eye-space $z_e>0$ and near/far $n,f$, the projection in Section 4.2 gives

$$z_{\mathrm{ndc}}={\displaystyle \frac{f+n}{n-f}}+{\displaystyle \frac{2fn}{(n-f)z_e}},d=\frac{1}{2}{z}_{\mathrm{ndc}}+\frac{1}{2}.$$

(28)

Since $\partial d/\partial z_e\propto 1/z_e^2$, precision concentrates near n; pushing n outward reduces z-fighting in VR [4].

4.3.3. Normals: Inverse-Transpose and TBN

For linear object transform $M\in {\mathbb{R}}^{3\times 3}$,

$$N^{\prime }\propto {\left(M^{-1}\right)}^{\top }N,N^{\prime }\leftarrow {\displaystyle \frac{N^{\prime }}{\parallel N^{\prime }\parallel }}.$$

(29)

With normal maps, decode tangent-space normal $n_t$ and map via orthonormal basis $T,B,N$:

$$n_{\mathrm{world}}=\left[TBN\right]n_t.$$

(30)

This eliminates shading skew under non-uniform scales and deformations [4].

4.3.4. Texture Filtering and MIP Selection

Given screen-space differentials $\partial u/\partial x$, $\partial u/\partial y$, $\partial v/\partial x$, $\partial v/\partial y$, GPUs choose

$$\lambda ={log}_2\left(max\left\{\sqrt{{\left(\frac{\partial u}{\partial x}\right)}^2+{\left(\frac{\partial v}{\partial x}\right)}^2},\sqrt{{\left(\frac{\partial u}{\partial y}\right)}^2+{\left(\frac{\partial v}{\partial y}\right)}^2}\right\}\right),$$

(31)

then apply bilinear/trilinear sampling;anisotropic/Elliptical Weighted Average (EWA) filters approximate the projected footprints ellipse for grazing angles [46,47].

4.3.5. Linear Color and Fast Fresnel

Lighting is computed in linear space; encode to display with the standard RGB (sRGB) transfer function only at output. A common GPU Fresnel approximation is Schlick’s

$$F(cos\theta )=F_0+(1-F_0){(1-cos\theta )}^5,$$

(32)

which closely tracks conductor/dielectric behavior at a fraction of the cost [48,49].

4.4. Neural and Differentiable Rendering: Models and Math

Neural/differentiable rendering provides learnable scene representations and differentiable image formation suitable for VR.

4.4.1. Neural Radiance and Density Fields

A scene is represented by a network

$$F_{\theta }(\mathbf{x},\mathbf{d})=\left(\sigma \left(\mathbf{x}\right),c(\mathbf{x},\mathbf{d})\right),$$

(33)

where $\mathbf{x}\in {\mathbb{R}}^3$ (meters), $\mathbf{d}\in {\mathbb{S}}^2$ (unit), $\sigma \ge 0$ has units $1/\mathrm{m}$ (attenuation per unit length), and $c\in {[0,1]}^3$ is RGB; $\theta$ are MLP weights [10]. Range conventions: $\sigma$ typically uses softplus to enforce nonnegativity; c uses sigmoid to stay in gamut.

4.4.2. Frequency Encodings (Positional/SIREN)

To mitigate spectral bias and resolve detail, inputs are lifted by positional encodings

$$\gamma \left(\mathbf{x}\right)=\left[sin\left(2^0\pi \mathbf{x}\right),cos\left(2^0\pi \mathbf{x}\right),\dots ,sin\left(2^{L-1}\pi \mathbf{x}\right),cos\left(2^{L-1}\pi \mathbf{x}\right)\right],$$

(34)

applied per-coordinate; $\gamma \left(\mathbf{d}\right)$ uses fewer bands due to smoother view dependence [10,50]. An alternative uses periodic activations (SIREN),

$${\mathbf{h}}_{\ell +1}=sin\left({\omega }_0{W}_{\ell }{\mathbf{h}}_{\ell }+{\mathbf{b}}_{\ell }\right),$$

(35)

which supplies a Fourier-like basis intrinsically [51]. Bandwidth note:larger L or ${\omega }_0$ increases representable frequencies but raises sampling demands.

4.4.3. Differentiable Volume Rendering

Along a camera ray $\mathbf{r}\left(t\right)=\mathbf{o}+t\mathbf{d}$,

$$C\left(\mathbf{r}\right)={\int }_{t_n}^{t_f}T\left(t\right)\sigma \left(\mathbf{r}\left(t\right)\right)c(\mathbf{r}\left(t\right),\mathbf{d})dt,T\left(t\right)=exp\left(-{\int }_{t_n}^t\sigma \left(\mathbf{r}\left(s\right)\right)ds\right).$$

(36)

$T\left(t\right)$ is transmittance (survival probability to depth t). With stratified samples $\left\{t_i\right\}$, ${\delta }_i=t_{i+1}-t_i$,

$$\widehat{C}\left(\mathbf{r}\right)=\sum _{i=1}^N\underset{T_i}{\underbrace{exp\left(-\sum _{j=1}^{i-1}{\sigma }_j{\delta }_j\right)}}\underset{{\alpha }_i}{\underbrace{\left(1-exp(-{\sigma }_i{\delta }_i)\right)}}{c}_i,$$

(37)

In Equation (36), $T_i=exp(-{\sum }_{j<i}{\sigma }_j{\delta }_j)$ is transmittance and ${\alpha }_i=1-exp(-{\sigma }_i{\delta }_i)$ is opacity, so $w_i=T_i{\alpha }_i$ yields front-to-back compositing along the ray [10]. Intuitively, higher density ${\sigma }_i$ increases opacity and attenuates contributions from later samples, which is the key mechanism enabling differentiable view synthesis in real-time VR pipelines.

Gradients: $\partial \widehat{C}/\partial c_i=w_i$; $\partial {\alpha }_i/\partial {\sigma }_i={\delta }_i{e}^{-{\sigma }_i{\delta }_i}$; autodiff handles the dependence of later $T_k$ ($k>i$) on earlier ${\sigma }_i$.

4.4.4. Differentiable Rasterization

(i) Forward model.

Vertices $X_w\in {\mathbb{R}}^3$ map to camera space $X_c=R{X}_w+t$ and to pixels via the pinhole projection

$$u=f_x{\displaystyle \frac{X_c}{Z_c}}+c_x,v=f_y{\displaystyle \frac{Y_c}{Z_c}}+c_y.$$

(38)

Shading produces color $S(u,v;{\theta }_{\mathrm{sh}})$ (BRDF/texture/lighting). The per-image loss is

$$\mathcal{L}=\sum _{\mathbf{p}}\rho \left(C\left(\mathbf{p}\right)-C^{*}\left(\mathbf{p}\right)\right),$$

(39)

with robust penalty $\rho (·)$ [13]. Purpose: Equations (38) and (39) define the forward pass and objective.

(ii) Projection Jacobians (pose). With $X_c={(X_c,Y_c,Z_c)}^{\top }$, the image Jacobian is

$$J_{\pi }\left(X_c\right)=\left[\begin{array}{ccc}{\displaystyle \frac{f_x}{Z_c}}& 0& -f_x{\displaystyle \frac{X_c}{Z_c^2}}\\ 0& {\displaystyle \frac{f_y}{Z_c}}& -f_y{\displaystyle \frac{Y_c}{Z_c^2}}\end{array}\right].$$

(40)

For an SE(3) twist $\delta \xi =(\delta t,\delta \varphi )$,

$${\displaystyle \frac{\partial X_c}{\partial \xi }}=\left[I-{\left[X_c\right]}_{\times }\right]\in {\mathbb{R}}^{3\times 6},{\displaystyle \frac{\partial (u,v)}{\partial \xi }}=J_{\pi }\left(X_c\right){\displaystyle \frac{\partial X_c}{\partial \xi }}.$$

(41)

Manifold-consistent updates keep $R\in \mathrm{SO}\left(3\right)$:

$$R\leftarrow Rexp\left({\left[\delta \varphi \right]}_{\times }\right),{\left[\delta \varphi \right]}_{\times }=\left[\begin{array}{ccc}0& -\delta {\varphi }_z& \delta {\varphi }_y\\ \delta {\varphi }_z& 0& -\delta {\varphi }_x\\ -\delta {\varphi }_y& \delta {\varphi }_x& 0\end{array}\right].$$

(42)

Purpose: Equations (40) and (42) give pose sensitivities and a stable pose update [13].

(iii) Shape and intrinsics derivatives. Linear shape bases $B_k$ give

$$X_w\left(\alpha \right)=X_w^{\left(0\right)}+\sum _k{\alpha }_k{B}_k,{\displaystyle \frac{\partial (u,v)}{\partial {\alpha }_k}}=J_{\pi }(R{X}_w+t)R{B}_k.$$

(43)

Intrinsics gradients are

$${\displaystyle \frac{\partial u}{\partial f_x}}={\displaystyle \frac{X_c}{Z_c}},{\displaystyle \frac{\partial v}{\partial f_y}}={\displaystyle \frac{Y_c}{Z_c}},{\displaystyle \frac{\partial u}{\partial c_x}}=1,{\displaystyle \frac{\partial v}{\partial c_y}}=1.$$

(44)

Purpose: Equations (43) and (44) provide shape/camera sensitivities.

(iv) Smooth visibility/occlusion. Soft coverage (edge SDF $s_i\left(\mathbf{p}\right)$) and soft z-buffer give nonzero gradients near silhouettes:

$$w_i\left(\mathbf{p}\right)=\sigma \left(-\kappa s_i\left(\mathbf{p}\right)\right),\sigma \left(z\right)={\displaystyle \frac{1}{1+e^{-z}}},\kappa >0,$$

(45)

$${\omega }_i\left(\mathbf{p}\right)={\displaystyle \frac{exp(-\tau z_i)}{{\sum }_{j}exp(-\tau z_j)}},\tau >0.$$

(46)

Blended color:

$$C\left(\mathbf{p}\right)=\sum _i\left(w_i\left(\mathbf{p}\right){\omega }_i\left(\mathbf{p}\right)\right)S_i(\mathbf{p};{\theta }_{\mathrm{sh}}).$$

(47)

Purpose: Equations (45)–(47) relax discrete visibility for differentiability (cf. OpenDR [13]).

(v) Shading derivatives (chain rule). For texture sampling with UVs $(u_t,v_t)$,

$${\displaystyle \frac{\partial S}{\partial X_c}}={\displaystyle \frac{\partial S}{\partial (u_t,v_t)}}{\displaystyle \frac{\partial (u_t,v_t)}{\partial (u,v)}}{\displaystyle \frac{\partial (u,v)}{\partial X_c}},$$

(48)

and analogously for $\xi ,{\alpha }_k,f_x,f_y,c_x,c_y$. Note: Include TBN for normal maps and any material channels in $\partial S/\partial (·)$.

(vii) Loss-gradient assembly. With residual $r\left(\mathbf{p}\right)=C\left(\mathbf{p}\right)-C^{*}\left(\mathbf{p}\right)$ and ${\rho }^{\prime }\left(r\right)$,

$${\displaystyle \frac{\partial \mathcal{L}}{\partial \theta }}=\sum _{\mathbf{p}}{\rho }^{\prime }\left(r\left(\mathbf{p}\right)\right)\left(\sum _i{\displaystyle \frac{\partial C}{\partial S_i}}{\displaystyle \frac{\partial S_i}{\partial \theta }}+\sum _i{\displaystyle \frac{\partial C}{\partial {\mathbf{u}}_i}}{\displaystyle \frac{\partial {\mathbf{u}}_i}{\partial \theta }}\right),$$

(49)

for $\theta \in \{\xi ,{\alpha }_k,f_x,f_y,c_x,c_y,{\theta }_{\mathrm{sh}}\}$. Note: $\frac{\partial C}{\partial {\mathbf{u}}_i}$ accumulates coverage and depth terms from Equations (45) and (46).

(viii) Stability practices (concise checklist).

• Coarse$\to$fine schedules: solve pose/intrinsics first, then shape/appearance.
• Clamp softness: choose $\kappa ,\tau$ to avoid vanishing/exploding gradients at edges.
• Use robust penalties $\rho$ and multi-view priors for regularization.
• For avatars, pair with learned face/body models for stronger priors [38,39,52].

4.4.5. Training Objective and Regularization

Given calibrated multi-view targets $C^{*}\left(\mathbf{r}\right)$,

$$\underset{\theta }{min}\sum _{\mathbf{r}\in \mathcal{R}}\parallel {\widehat{C}}_{\theta }\left(\mathbf{r}\right)-C^{*}\left(\mathbf{r}\right){\parallel }_2^2+\lambda {\mathcal{L}}_{\mathrm{reg}}\left(\theta \right),$$

(50)

where common regularizers include sparsity on $\sigma$ and distortion/smoothness on accumulated density to reduce floaters [10]. Differentiability: the loss backpropagates through sampling, transmittance, and network layers by the chain rule.

4.4.6. Acceleration for Interactive VR

To approach dual-eye 90/120 Hz, systems (i) factorize fields into many tiny MLPs for parallel evaluation [11], (ii) amortize multi-bounce lighting via neural radiance caching inside real-time path tracing [12], and (iii) reduce memory pressure in large scenes through streamable, memory-efficient radiance field representations that support partitioning and on-demand streaming [53]. Budgeting tip: couple per-eye foveation with coarser sampling/offline caches in the periphery to respect VR frame budgets. In practice, learning-based accelerations should be paired with runtime monitoring and classical fallbacks to remain robust under latency spikes and out-of-distribution content (Section 6.8).

4.5. Hybrid and Perceptual Rendering for VR

VR must sustain high frame rates with limited budgets. Perceptual strategies modulate computation where the human visual system is most sensitive. Foveated rendering concentrates resolution and shading near the gaze point, guided by eye tracking; recent surveys organize algorithms and hardware support [16,17,19]. Temporal reprojection/upsampling and dynamic level-of-detail further trade accuracy for stability.

Hybrid renderers mix rasterization (for primary visibility and nearby geometry) with selective ray/path tracing (for glossy reflections, soft shadows, GI), allocating work to regions that most impact perceived quality. Perceptual error metrics and attention-aware policies help decide when to refine shading or geometry [18,22]. Together, these trends show a shift from purely deterministic shading to hybrid, learned, and perceptually guided rendering, tailored to the stringent constraints of immersive VR.

4.6. Summary of Section 4

Section 4 reviewed how classical shading, camera and projection models, GPU math, and newer neural or differentiable methods are combined into practical VR rendering pipelines.

Table 5. Section 4 at-a-glance: rendering foundations and modern extensions for VR. The summary spans physically based shading, camera and projection models for stereo, GPU-precision considerations, and neural/differentiable methods for view synthesis. Perceptual strategies such as foveation allocate computation where users are most sensitive, supporting real-time performance without sacrificing perceived quality.

Topic (Section)	Core Math/Formulation	VR Impact
Classical shading (Section 4.1)	Physically based microfacet BRDFs with energy-conserving parameters; Schlick-style Fresnel; compute lighting in linear space and apply the sRGB transfer function only at the final output.	Gives physically plausible highlights and stable tone reproduction at real-time rates.
Cameras & projection (Section 4.2)	Per-eye off-axis frusta from IPD and lens geometry using $(l,r,b,t)$; lens pre-distortion; orientation- and depth-aware timewarp/spacewarp that reuse the same per-eye projection as the main render.	Produces correct stereo geometry and reduces apparent latency without resubmitting full frames.
GPU math essentials (Section 4.3)	Perspective-correct interpolation realised by dividing attributes by clip-space w, interpolating, then renormalizing; depth precision concentrated near the near plane; normal transform via the inverse-transpose of the model matrix; MIP level estimated from screen-space derivatives, with anisotropic or EWA filtering when needed.	Avoids affine warping, mitigates z-fighting by pushing the near plane, fixes shading skew, and preserves texture detail at grazing angles.
Neural & differentiable rendering (Section 4.4)	Neural radiance fields for view synthesis with volumetric integration along rays; differentiable rasterization that exposes derivatives with respect to pose and intrinsics, updating rotations on the $\mathrm{SO}\left(3\right)$ manifold.	Enables gradient-based tracking, relocalization, and online calibration; real-time use becomes feasible via network factorization and radiance caching.
Hybrid & perceptual (Section 4.5)	Eye-tracked foveated rendering; temporal reprojection and upsampling; hybrid pipelines where rasterization handles primary visibility while selected reflections, soft shadows, and global illumination use ray or path tracing.	Concentrates computation where perception is most sensitive and improves quality-per-watt under 90/120 Hz stereo constraints.

summarizes the core ideas and their impact on VR systems.

Practical Takeaways

• Do all shading computations in linear space, and apply the sRGB transfer function only once at the final color write.
• Use the same per-eye off-axis projection in both the main render and the timewarp/spacewarp paths to avoid extra distortion.
• Push the near plane outward as far as content allows, and transform normals with the inverse-transpose of the model matrix to reduce depth and shading artifacts.
• In NeRF-style or neural field training, make physical units explicit (for example, treat density as “per meter”) and monitor the accumulated opacity weights that actually scale image residuals and gradients.
• For 90/120 Hz stereo, allocate resolution and shading budget to the fovea, and rely on cheaper sampling, models, or caches in the periphery.

5. Interaction Modeling and User Interface Dynamics

User interaction in VR is a closed-loop process spanning sensing, state estimation, kinematic/physics modeling, rendering, and human motor response. This section consolidates foundational models for pose and manipulation, then organizes advance during the recent five years (2021–2025) that fuse learning-based perception, differentiable optimization, and perceptual constraints into the interaction loop.

5.1. Foundations: Sensing, Pose, and Kinematic Mapping

This subsection formalizes the sensing→pose pipeline and the kinematic mapping used to drive hands, tools, and avatars in VR under real-time constraints. Contemporary VR interaction follows the classic 3D UI taxonomy (selection, manipulation, travel, system control) [15]. Rigid-body poses evolve on $\mathrm{SE}\left(3\right)$; orientations are commonly represented by unit quaternions for numerically stable interpolation and composition [7,34,35]. End-effector goals (e.g., a hand or tool tip) are mapped to joint angles via FK/IK with joint limits and comfort costs [8].

Foundational work on modeling and controlling virtual humans provides enduring principles for avatar kinematics, posture control, and animation pipelines, complementing the FK/IK formulations used in VR embodiment [54].

Social VR further couples embodiment with multisensory feedback. For example, even simple collision haptics during human–virtual human interaction can measurably influence presence and user experience, motivating careful feedback design alongside stability constraints [55]. In practice, low-latency controllers rely on Jacobian-based updates.

(i)

Problem setup and notation. Given a task-space target $x^{\star }\in {\mathbb{R}}^k$ and forward kinematics $f\left(\theta \right)\in {\mathbb{R}}^k$ with joint angles $\theta \in {\mathbb{R}}^m$, define the task error

$$e=x^{\star }-f\left(\theta \right).$$

(51)

Here $k\in \{3,6\}$ denotes position/pose, and m is the number of degrees of freedom.

(ii)

Geometric Jacobian. With the geometric Jacobian $J=\partial f/\partial \theta \in {\mathbb{R}}^{k\times m}$, small variations satisfy the linear approximation

$$\delta x\approx J\delta \theta .$$

(52)

In real time, J is updated numerically or from analytic expressions.

(iii)

DLS (damped least-squares) update. To remain stable near singularities, use the damped update

$$\Delta \theta =\eta J^{\top }{\left(J{J}^{\top }+{\lambda }^{2}I\right)}^{-1}e,$$

(53)

Here the damping term ${\lambda }^{2}I$ regularizes the inverse near singular configurations by keeping $(J{J}^{\top }+{\lambda }^{2}I)$ well-conditioned, so $\Delta \theta$ remains bounded even when J is ill-conditioned. In VR hand/avatar control, this trades accuracy for stability under tight frame budgets, and $\lambda$ is typically tuned to avoid jitter while maintaining responsive convergence.

(iv)

Task weighting and conditioning. If task components have different priorities, introduce $W\succ 0$:

$$\Delta \theta =\eta J^{\top }{\left(J{J}^{\top }+{\lambda }^{2}W\right)}^{-1}e,$$

(54)

which improves conditioning and allows per-axis tolerances.

(v)

Secondary objectives via null space. To address posture comfort or joint-limit avoidance without disturbing the primary task,

$$\Delta \theta \leftarrow \Delta \theta +\left(I-J^{+}J\right)\left(-{\nabla }_{\theta }C\left(\theta \right)\right),$$

(55)

where $C\left(\theta \right)$ is a secondary cost and $J^{+}$ the pseudoinverse.

(vi)

Lightweight alternatives. When matrix solves are too expensive, use the Jacobian-transpose heuristic

$$\Delta \theta =\eta J^{\top }e$$

(56)

or cyclic coordinate descent (CCD). These are approximations but very fast.

(vii)

Closing the loop (tracking and full-body drive). Visual–inertial (inside-out) tracking estimates head/hand 6DoF poses; SLAM back-ends stabilize world anchors over long interactions [40]. For detailed hands/faces, learned regressors provide strong priors for retargeting and animation [39]. Empirically, controller input can outperform vision-only hand tracking on some tasks, motivating context-aware switching policies [56].

5.2. Hand, Body, and Object Interaction

Direct manipulation (grasp, pinch, poke) hinges on accurate articulated tracking with temporal smoothing and constraint projection to suppress jitter near contact [57]. Whole-body interaction blends controller priors, optical cues, and IK to maintain reachability and avoid self-collision in tight spaces [58]. Robust contact and proximity tests rely on efficient collision detection, broad-phase culling, and convex distance queries (e.g., GJK), which also inform haptic proxies and constraint solvers [26,28].

A growing trend is controller-less tracking from headset-mounted egocentric cameras, where self-occlusion and motion blur become dominant failure modes. Large-scale egocentric datasets enable such body/hand tracking under realistic occlusions, strengthening avatar embodiment pipelines [59].

5.3. Constraint-Based Manipulation and Differentiable Calibration

This subsection formalizes contact handling for VR manipulation at real-time rates using three complementary views–impulse/velocity, convex QP, and position-level projection–and then shows how differentiable sensing/graphics enable online calibration.

5.3.1. Impulse-/Velocity-Level Contact Resolution

Let $q,v$ be configuration and velocity, $M\succ 0$ the generalized mass, J the contact Jacobian, and $f_{\mathrm{ext}}$ external forces. Over a step $\Delta t$,

$$M(v_{t+1}-v_t)=\Delta t{f}_{\mathrm{ext}}+J^{\top }\lambda ,$$

(57)

where $\lambda$ are contact impulses. With normal gap $g_n\left(q\right)\ge 0$ (signed distance), nonpenetration is enforced by complementarity

$$g_n\left(q_{t+1}\right)\ge 0,{\lambda }_n\ge 0,{\lambda }_n^{\top }{g}_n\left(q_{t+1}\right)=0,$$

(58)

and tangential (Coulomb) friction by a cone $\parallel {\lambda }_t\parallel \le \mu {\lambda }_n$ or a friction pyramid. A velocity-level variant uses

$$J_n{v}_{t+1}+b_n\ge 0,{\lambda }_n\ge 0,{\lambda }_n^{\top }\left(J_n{v}_{t+1}+b_n\right)=0,$$

(59)

with $b_n$ collecting Baumgarte/restition terms. These yield a mixed complementarity problem (MCP) in $\lambda$ (or in $(v_{t+1},\lambda )$) [2,21].

5.3.2. Convex QP Time Stepping

A common real-time scheme solves a convex projection in velocity space:

$$\underset{\Delta v,\lambda }{min}{\displaystyle \frac{1}{2}}{\parallel \Delta v-\Delta v_{\mathrm{free}}\parallel }_M^2\mathrm{subject}\mathrm{to}{J}_n\Delta v+b_n\ge 0,\mathrm{friction}\mathrm{constraints},$$

(60)

where $\Delta v_{\mathrm{free}}=M^{-1}\Delta t{f}_{\mathrm{ext}}$. Stationarity gives the KKT condition $M\Delta v-J^{\top }\lambda =0$; together with the inequalities and the complementarity

$${\lambda }_n^{\top }\left(J_n\Delta v+b_n\right)=0,$$

(61)

(60) is equivalent to an LCP when a friction pyramid is used, or to a SOCP with a circular cone $\parallel {\lambda }_t\parallel \le \mu {\lambda }_n$. This framing unifies projected Gauss–Seidel/cone-projection and interior-point solvers under one objective [2].

5.3.3. Geometric (Position-Level) Projection

Position-level methods correct constraint violations directly:

$$\underset{\Delta q}{min}{\displaystyle \frac{1}{2}}{\parallel \Delta q\parallel }^2\mathrm{subject}\mathrm{to}C(q+\Delta q)=0,$$

(62)

where $C\left(q\right)=0$ encodes joints/contacts. Linearizing,

$$\Delta q\approx -J_C^{\top }{\left(J_C{J}_C^{\top }\right)}^{-1}C\left(q\right),$$

(63)

with $J_C=\partial C/\partial q$. In practice, constraints are processed iteratively (Gauss–Seidel) within the frame budget; small compliance or damping in C regularizes contact jitter [2].

5.3.4. Differentiable Calibration and Identification

Online calibration estimates unknown frames/offsets (e.g., controller/tool pose offsets or contact frames) by minimizing visual reprojection or photometric residuals through differentiable graphics/kinematics. With calibrated intrinsics K, 3D features $X_i$, image points $u_i$, and pose $T\in \mathrm{SE}\left(3\right)$,

$$\underset{T,\varphi }{min}\sum _i\parallel \pi \left(KT\left(\varphi \right)X_i\right)-u_i{\parallel }_2^2+\lambda {\mathcal{L}}_{\mathrm{photo}},$$

(64)

where $\varphi$ parameterizes the pose increment and $\pi$ is projection. Updates are applied on the manifold,

$$T\leftarrow TExp\left(\widehat{\varphi }\right),$$

(65)

which preserves group structure and improves stability [13]. For contact-rich skills, differentiable physics adds residuals on contact consistency (e.g., nonpenetration, frictional stick/slide), enabling recovery of latent states/forces from video or tactile surrogates and improving data efficiency in policy learning [30].

5.4. Haptics and Force-Feedback for Action–Perception Coupling

Haptic channels tighten the loop by reflecting contact and stiffness cues. Lightweight devices and exoskeletons typically use PID/impedance targets with passivity-minded gains to remain stable under network and rendering latency; classic PHANToM-style designs and control envelopes are well documented [60,61]. A canonical proxy-based approach is the constraint-based god-object formulation, which enforces non-penetration while maintaining stable force feedback [62]. In soft-tissue or tool-mediated tasks, reduced-order FEM or proxy models support high-rate (kHz) force updates while the visual path runs at display refresh, sustaining stability and realism [63].

Discrete-Time

Passivity (PO/PC) Let $f_h\left[k\right],v_h\left[k\right]$ be device-port force/velocity at sample k and $T_s$ the servo period. A passivity observer tracks energy

$$E[k+1]=E\left[k\right]+T_s{f}_h{\left[k\right]}^{\top }{v}_h\left[k\right]-T_s{f}_v{\left[k\right]}^{\top }{v}_h\left[k\right],E\left[0\right]\ge 0,$$

(66)

where $f_v$ is the virtual environment force. If $E[k+1]<0$ (incipient non-passivity), a passivity controller injects adaptive damping

$$f_v\left[k\right]\leftarrow f_v\left[k\right]+\alpha \left[k\right]v_h\left[k\right],\alpha \left[k\right]={\displaystyle \frac{-E[k+1]}{T_s{\parallel v_h\left[k\right]\parallel }^2+\epsilon }},$$

(67)

Equation (65) monitors the net energy exchanged at the device port; if $E[k+1]<0$, the coupled loop is injecting energy (non-passive), which can trigger unstable oscillations under delay or stiff virtual contacts. Equation (66) restores passivity by adding the minimum adaptive damping $\alpha \left[k\right]v_h\left[k\right]$ needed to dissipate the excess energy, making kHz haptic updates stable while visuals run at lower refresh rates.

Which guarantees $E[k+2]\ge 0$ and restores passivity while minimally perturbing the nominal interaction. Energy-tank variants enforce $E\left[k\right]\in [0,E_{\max }]$ to budget aggressive transients (tool impacts) before throttling feedback. These controllers are compatible with the proxy/FEM pipelines above and preserve stability across rendering-latency fluctuations [60,61,63]. Comparative studies further show that visuo-haptic versus visual-only feedback can systematically change presence, underscoring that modeling priorities depend on the dominant feedback channel [64].

5.5. Perception-Aware Interaction: Foveation, Workload, and Cybersickness

Perceptual limits bound what must be simulated for believable action. Eye-tracked foveation concentrates shading and sampling near the gaze point, freeing budget for physics/IK during action onsets; algorithmic trade-offs and system design are summarized in recent surveys [16,17,19]. Perceptual LOD throttles distant/occluded dynamics without degrading agency. For comfort, ML-based cybersickness predictors leverage demographics and behavioral/physiological signals; Personalized cybersickness prediction models can improve forecasting accuracy at the individual level, enabling comfort-aware adaptation policies in real time [65]. These models can drive online mitigation (e.g., locomotion gains, vignette strength) during interaction [66,67,68].

5.6. Learning-Augmented Interfaces

Interfaces increasingly fuse geometric cores with learned scene/function representations. NeRF-style encodings assist occlusion-aware queries and grasp target prediction; factorized fields accelerate spatial lookups for interaction [10,11]. Graph/mesh operators enable spatial reasoning over layouts (affordances, collision margins) in shared spaces [36,37]. End-to-end policies co-train perception and control through differentiable kinematics/rendering losses for low-drift hand–object tasks [39]. Personalized, animatable avatars reduce retargeting error and improve ownership in multi-user scenes [31,52].

Emerging LLM-driven agents in social VR suggest a path toward more natural dialogue and interactive behavior; however, they introduce new constraints on latency, safety, and evaluation [69]. These trends motivate future benchmarking that jointly measures interaction quality and real-time system performance.

Despite their promise, learning-augmented interfaces should be paired with deployment guardrails (latency margins, validation, fallbacks) to remain robust in diverse real-time VR settings (Section 6.8).

5.7. Robustness, Latency, and System Practices

Interaction quality depends on stable 6DoF state, contact resolution, and motion-to-photon latency. Practical engines prioritize foreground constraints, defer background dynamics, and align sensing/physics/render clocks to minimize jitter [15]. Physics choices (semi-implicit/symplectic steps for energy behavior, adaptive $\Delta t$ for contact bursts) directly influence grasp stability and selection accuracy [2].

5.8. Summary of Section 5

Section 5 collected the main ingredients for physically grounded, yet comfortable VR interaction: manifold-aware pose updates, IK for arm and hand control, contact and constraint handling, online calibration, and perceptual allocation of computation and haptics. The main design rules are:

(i)

Maintain poses on $\mathrm{SE}\left(3\right)$ with quaternion/Lie updates; fuse visual–inertial sensing to stabilize world-locked anchors.

(ii)

Map end-effector goals with DLS/transpose/CCD under joint limits; use null-space terms to encourage comfort and good posture.

(iii)

Resolve contacts via impulse/QP or position-level projection, iterating within the frame budget and regularizing to suppress jitter.

(iv)

Calibrate online through differentiable kinematics and rendering with manifold-consistent pose updates.

(v)

Allocate work perceptually (eye-tracked foveation, perceptual LOD); keep haptics passive under latency using passivity observer/controller (PO/PC) style damping.

As a compact reference for reporting and reproducible evaluation,

Table 6. Interaction metrics glossary for Section 5. The listed metrics quantify manipulation accuracy, contact feasibility, calibration quality, and responsiveness in ways that are directly measurable in VR experiments. Units and measurement contexts are included to support reproducible reporting, and the associated formulas provide precise definitions used later in benchmarks and case studies (Section 6).

Metric	Symbol	What It Measures	Notes/Units
EE position RMSE	Equation (68)	Average Euclidean error of the end-effector position relative to a reference trajectory.	Meters, in the world frame.
EE orientation error	Equation (69)	Angular misalignment between predicted and reference orientations (geodesic angle on $\mathrm{SO}\left(3\right)$).	Degrees or radians.
Contact infeasibility	Equation (70)	Largest remaining penetration depth over all active contacts.	Meters; should be close to zero.
Complementarity residual	Equation (71)	Violation of the normal-contact complementarity $0\le {\lambda }_n\perp g_n\ge 0$.	Impulse·meter; zero means perfectly feasible contacts.
Reprojection RMSE	Equation (72)	Pixel-wise error when reprojecting tracked visual features into the image.	Pixels in the image plane.
Calibration drift	Equation (73)	Accumulated change of the estimated pose on $\mathrm{SE}\left(3\right)$ over time.	Geodesic distance, e.g., $\parallel log(\Delta T)\parallel$.
Control latency	Equation (74)	Delay from sensing to the corresponding actuation reaching the user.	Milliseconds.
Slip rate	Equation (75)	How often a grasp unintentionally fails during interaction.	Fraction in $[0,1]$ (events per grasp).
Passivity margin	Equations (76) and (77)	Energy balance in the haptic loop (how far the system is from violating passivity).	Nonnegative is desired; larger margins are safer.

summarizes the interaction metrics used throughout Section 5.

Reporting Metrics

Notation

${\widehat{\mathbf{x}}}_i$ (prediction), ${\mathbf{x}}_i^{*}$ (reference); $\angle \left(R\right)=arccos\left(\right(trR-1)/2)$; $g_{+}\left(q\right)=max(g\left(q\right),0)$; ⊙ denotes Hadamard product; $\pi (·)$ is the projection with intrinsics K; $\parallel \Delta T\parallel ={\parallel log(\Delta T)\parallel }_2$ is the geodesic norm on $\mathrm{SE}\left(3\right)$.

$$\mathbf{EE}\mathbf{position}\mathbf{RMSE}=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N\parallel {\widehat{\mathbf{x}}}_i-{\mathbf{x}}_i^{*}{\parallel }_2^2}$$

(68)

$$\mathbf{EE}\mathbf{orientation}\mathbf{error}=\angle \left({\widehat{R}}_i{R}_i^{*\top }\right)$$

(69)

$$\mathbf{Contact}\mathbf{infeasibility}=\parallel g_{+}\left(q\right){\parallel }_{\infty }$$

(70)

$$\mathbf{Complementarity}\mathbf{residual}=\parallel {\lambda }_n\odot g_n\left(q\right){\parallel }_{\infty }$$

(71)

$$\mathbf{Reprojection}\mathbf{RMSE}=\sqrt{{\displaystyle \frac{1}{\left|\mathcal{P}\right|}}\sum _{p\in \mathcal{P}}\parallel {\mathbf{u}}_p-\pi \left(K{\widehat{T}}_p{\mathbf{X}}_p\right){\parallel }_2^2}$$

(72)

$$\mathbf{Calibration}\mathbf{drift}=\parallel \Delta T\parallel =\parallel log\left({\widehat{T}}_t^{-1}{T}_t\right){\parallel }_2$$

(73)

$$\mathbf{Control}\mathbf{latency}=t_{\mathrm{pose}\mathrm{out}}-t_{\mathrm{sensor}\mathrm{in}}$$

(74)

$$\mathbf{Slip}\mathbf{rate}={\displaystyle \frac{\#\left\{\mathrm{unintended}\mathrm{releases}\right\}}{\#\left\{\mathrm{grasps}\right\}}}$$

(75)

$$\begin{array}{cc}\hfill E[k+1]& =E\left[k\right]+T_s{f}_h{\left[k\right]}^{\top }{v}_h\left[k\right]-T_s{f}_v{\left[k\right]}^{\top }{v}_h\left[k\right],\hfill \\ \hfill \mathbf{Passivity}\mathbf{margin}& =\underset{0\le j\le k}{min}E\left[j\right](\ge 0\mathrm{desired}).\hfill \end{array}$$

(76)

$$\mathrm{Enforce}\mathrm{passivity}\mathrm{by}:f_v\left[k\right]\leftarrow f_v\left[k\right]+\alpha \left[k\right]v_h\left[k\right],\alpha \left[k\right]={\displaystyle \frac{-min(0,E[k+1\left]\right)}{T_s{\parallel v_h\left[k\right]\parallel }^2+\epsilon }}.$$

(77)

6. Future Direction of Mathematical Modeling in VR

This section charts near-term (1–2 year) opportunities for VR that are (i) geometry-aware–numerics and estimation on appropriate manifolds with constraint/contact handling [2,5,21], (ii) perception-aware–budgets and error bounded by human sensitivity via foveation and comfort models [16,17,19,22], and (iii) learning-augmented–neural modules used where they offer the largest cost–benefit while preserving structure [10,11,12,13,14,40]. Subsections detail: physics and time stepping (Section 6.1), spatial estimation/SLAM (Section 6.2), neural/differentiable rendering (Section 6.3), avatars and IK (Section 6.4), haptics and collision (Section 6.5), and cybersickness-aware scheduling (Section 6.6). Common metrics/targets and a concise reporting template appear in Section 6.7.

6.1. Learning-Augmented Physics and Stable Time Stepping

6.1.1. Structure-Preserving Time Stepping

Let $(q,p)$ be generalized coordinates and momenta with Hamiltonian $H(q,p)=T\left(p\right)+V\left(q\right)$ and holonomic constraints $c\left(q\right)=0$ with Jacobian $G=\partial c/\partial q$. A symplectic Euler step with multipliers $\lambda$ is

$$\begin{array}{cc}\hfill p_{k+1}& =p_k-\Delta t\nabla V\left(q_k\right)-G{\left(q_k\right)}^{\top }{\lambda }_k,\hfill \\ \hfill q_{k+1}& =q_k+\Delta t{M}^{-1}{p}_{k+1},c\left(q_{k+1}\right)=0,\hfill \end{array}$$

(78)

which preserves a discrete symplectic form and yields bounded long-horizon energy drift when constraint residuals are controlled [2,20]. Here $M\succ 0$ is the mass matrix and ${\lambda }_k$ enforces $c\left(q_{k+1}\right)=0$.

6.1.2. Learning Insertion Without Breaking Structure

A learned module predicts parameters ${\varphi }_{\theta }$ and a conservative correction $U_{\theta }\left(q\right)$ so that $F_{\theta }\left(q\right)=-\nabla U_{\theta }\left(q\right)$, replacing $V\leftarrow V+U_{\theta }$ while keeping the step symplectic. If only a black-box force ${\tilde{F}}_{\theta }$ is available, project it to a conservative field via a Poisson solve

$${\psi }_{\theta }=arg\underset{\psi }{min}{\int }_{\mathrm{\Omega }}{\parallel \nabla \psi \left(q\right)-{\tilde{F}}_{\theta }\left(q\right)\parallel }^{2}dq,F_{\theta }\leftarrow \nabla {\psi }_{\theta },$$

(79)

so the update still derives from a potential [14,32,33].

6.1.3. Nonsmooth Contact and Friction

Normal complementarity and frictional cones (or pyramids) act on impulses $\lambda$:

$$g_n\left(q_{k+1}\right)\ge 0,{\lambda }_n\ge 0,{\lambda }_n^{\top }{g}_n\left(q_{k+1}\right)=0,\parallel {\lambda }_t\parallel \le \mu {\lambda }_n,$$

(80)

yielding an MCP/LCP/SOCP depending on the friction model [2,21]. Here $g_n$ is signed gap, $\mu$ the friction coefficient, and $({\lambda }_n,{\lambda }_t)$ the normal/tangential impulses.

6.1.4. Stability and What to Report

If ${\nabla }^2(V+U_{\theta })⪯LI$ and constraints are projected each step, discrete Grönwall bounds give

$$|H(q_k,p_k)-H(q_0,p_0)|\le C\Delta t\sum _{j<k}\left(\parallel c\left(q_j\right)\parallel +\mathrm{friction}\mathrm{viol}.\right),$$

(81)

so energy drift is controlled by residuals [2,20]. Report energy drift (%), max nonpenetration/equality violation, and fallback/substep rate as in Section 6.7 [18,22].

6.2. Spatial Estimation on Manifolds and Differentiable SLAM

6.2.1. Factor-Graph Objective on $\mathrm{SE}\left(3\right)$

Given relative pose measurements $Z_{ij}$ and per-pixel matches $(u_{ip},X_p)$, estimate poses $T_i\in \mathrm{SE}\left(3\right)$, depths/structure D by

$$\underset{\left\{T_i\right\},D}{min}\sum _{(i,j)}\parallel {\Sigma }_{ij}^{-{\displaystyle \frac{1}{2}}}log\left(Z_{ij}^{-1}{T}_i^{-1}{T}_j\right){\parallel }_2^2+\sum _{(i,p)}\rho \left(\parallel u_{ip}-\pi \left(K{T}_i{X}_p\left(D\right)\right){\parallel }_2^2\right),$$

(82)

where $log(·)$ is the Lie log on $\mathrm{SE}\left(3\right)$ and $\rho$ a robustifier. This unifies geometric (between-frames) and photometric (reprojection) terms [40].

6.2.2. On-Manifold Updates and Gauge

Optimize with $T_i\leftarrow T_{i}exp\left({\widehat{\xi }}_i\right)$ to remain on $\mathrm{SE}\left(3\right)$. Fix one pose (and global scale for monocular) to remove gauge freedoms; otherwise the normal equations are rank-deficient [5,40].

6.2.3. Metrics and Alignment Choices

Use ATE/RPE and per-frame reprojection RMSE with alignment stated explicitly (Sim(3) for monocular; SE(3) for stereo/RGB-D), following Section 6.7 [5,40].

6.3. Neural and Differentiable Rendering for Real-Time XR

6.3.1. Cached Estimators and Perceptual Budgeting

For per-pixel radiance, combine a fresh unbiased estimate ${\tilde{L}}^{\left(t\right)}$ and a cached running estimate ${\widehat{L}}^{(t-1)}$ by

$${\widehat{L}}^{\left(t\right)}=\alpha {\tilde{L}}^{\left(t\right)}+(1-\alpha ){\widehat{L}}^{(t-1)},$$

(83)

whose MSE decomposes into bias $\propto {(1-\alpha )}^2$ and variance $\propto {\alpha }^2/N$. Choose $(\alpha ,N)$ by minimizing samples under a CSF-derived weight $w(\theta ,f)$ such that $\mathrm{MSE}·w\le {\epsilon }_{\mathrm{JND}}$ [16,19]. Neural radiance caching and tiny-MLP factorization supply ${\tilde{L}}^{\left(t\right)}$ at VR rates [10,11,12].

6.3.2. Foveated Sampling as a Constrained Program

Let $\mathrm{\Lambda }\left(\mathbf{x}\right)$ be spp and $w\left(\mathbf{x}\right)$ the gaze-weight. Solve

$$\underset{\mathrm{\Lambda }\in \mathbb{N}}{min}\sum _{\mathbf{x}}\mathrm{\Lambda }\left(\mathbf{x}\right)\mathrm{subject}\mathrm{to}\mathrm{MSE}\left(\mathrm{\Lambda }\left(\mathbf{x}\right)\right)w\left(\mathbf{x}\right)\le \epsilon ,$$

(84)

to allocate samples where the HVS is most sensitive [16,19].

6.4. Avatars, IK, and Differentiable Bodies

6.4.1. Shape–Pose Estimation with Physics Priors

Estimate shape $\beta$, pose $\theta$, and global T by

$$\underset{\beta ,\theta ,T}{min}\sum _i\parallel u_i-\pi \left(KT{X}_i(\beta ,\theta )\right){\parallel }_2^2+{\lambda }_{\mathrm{contact}}{\parallel g{\left(q\left(\theta \right)\right)}_{+}\parallel }_2^2+{\lambda }_{\mathrm{prior}}R(\beta ,\theta ),$$

(85)

where $g(·)$ penalizes interpenetration and R encodes learned priors. This couples differentiable IK/rendering for stable personalization [38,39,52,70]. Here $X_i$ is the skinned vertex, $q\left(\theta \right)$ joint angles, and $u_i$ image points.

6.4.2. Identifiability and Sensing

Local identifiability requires the stacked Jacobian w.r.t. $(\beta ,\theta ,T)$ to have full column rank after gauge fixing; with HMD+controllers, regularization via R and contact terms reduces ambiguity [39,52].

6.5. Haptics, Collision, and Contact Modeling

6.5.1. Discrete Passivity with PO/PC

With sample period $T_s$ and port variables $(f_h\left[k\right],v_h\left[k\right])$, passivity requires

$$\sum _{k=0}^{N-1}{T}_s{f}_h{\left[k\right]}^{\top }{v}_h\left[k\right]\ge -E_0.$$

(86)

A passivity controller injects adaptive damping

$$f_v\left[k\right]\leftarrow f_v\left[k\right]+\alpha \left[k\right]v_h\left[k\right],\alpha \left[k\right]={\displaystyle \frac{max(0,-E[k+1\left]\right)}{T_s{\parallel v_h\left[k\right]\parallel }^2+\epsilon }},$$

(87)

to restore $E\left[k\right]\ge 0$ while minimally altering the virtual interaction [60,61,63]. Here $f_v$ is the virtual environment force and $E\left[k\right]$ the observed stored energy.

6.5.2. Deterministic Collision Bounds

For convex $A,B$, GJK/EPA yields $d(A,B)={min}_{a\in A,b\in B}{\parallel a-b\parallel }_2$ with BVH LOD. Guarantee $|d-\tilde{d}| \le \delta$ by bounding node radii, giving predictable haptic-step timing and stable force updates [26,28].

6.6. Cybersickness-Aware Scheduling and Perceptual Control

Predict-and-Allocate (MPC View)

Let a learned proxy ${\widehat{s}}_t=h_{\phi }\left(x_t\right)$ predict instantaneous sickness from signals $x_t$ (gaze, motion, frame-time, head kinematics). Allocate budgets $b_t=(b_t^{\mathrm{render}},b_t^{\mathrm{phys}},b_t^{\mathrm{anim}})$ by

$$\underset{b_t\ge 0}{min}{J}_t=\alpha {\widehat{s}}_t+\beta {\mathrm{MTP}}_t^2+\gamma {\mathrm{miss}}_t,\mathrm{subject}\mathrm{to}{b}_t^{\top }\mathbf{1}\le B.$$

(88)

with horizon-H MPC and device budget B. Train $h_{\phi }$ on SSQ proxies and behavioral/physiological features [66,67,68] and gate foveation/LOD as in [16,17]. Here ${\mathrm{MTP}}_t$ is motion-to-photon, and ${\mathrm{miss}}_t$ the missed-frame ratio.

6.7. Summary of Section 6

6.7.1. Guiding Principles

(i)

Preserve geometric structure in integration and estimation (symplectic/semi-implicit updates; on-manifold pose updates).

(ii)

Use learned priors/modules to cut cost, while keeping constraints/guarantees in the outer loop.

(iii)

Balance quality and performance with perceptual limits (eye-tracked/foveated budgets; just-noticeable thresholds).

To facilitate reproducible evaluation and cross-paper comparison, we summarize a concise set of common VR metrics in

Table 7. Common metrics across physics, estimation, rendering/systems, and comfort for VR evaluation. Physics-oriented measures track stability and constraint satisfaction; estimation measures capture tracking drift and consistency; system measures capture missed-frame behavior and latency; and comfort measures summarize user impact. Together, they provide a compact, reproducible checklist for reporting and cross-paper comparison.

Domain	Metric (Symbol)	What It Measures	Units/Notes
Physics	Energy drift (Equation (89))	Change of total energy over a window	% (lower is better)
	Constraint infeasibility (Equation (90))	Residuals for non-penetration/equality/velocity-level constraints	m or unitless
Estimation	ATE (Equation (91))	Absolute trajectory error after alignment	m (RMSE)
	RPE (Equation (92))	Relative pose error over gap $\Delta$	m/deg (RMSE)
	Reprojection RMSE (Equation (93))	Pixel-space residual per frame	pixels
Rendering/Systems	Missed-frame ratio (Equation (94))	Budget violations per frame	fraction (0–1)
	Visual rate (P95)	95th-percentile achieved display rate (or equivalently, 95th-percentile frame time)	Hz (higher is better); report alongside missed-frame ratio
	Motion-to-photon (MTP)	Sensor-to-photon latency (median/p95)	ms
	Per-eye throughput	Sustained res. and refresh	e.g., $2\times (2160 \times 2160)$ @ 90/120 Hz
Comfort	SSQ (Total/subscales)	Simulator Sickness Questionnaire	scalar scores
	Time-to-discomfort (Equation (95))	Exposure time until threshold	s or min
	Dropout rate (Equation (96))	Participants unable to complete exposure	fraction (0–1)

.

6.7.2. Definitions and Formulas

Notation

$\angle \left(R\right)=arccos\left(\right(trR-1)/2)$; $g_{+}\left(q\right)=max(g\left(q\right),0)$; $\pi (·)$ is the projection with intrinsics K; $\mathrm{RMSE}(·)$ denotes root-mean-square error; $S\in \mathrm{SE}\left(3\right)$ or $\mathrm{Sim}\left(3\right)$ aligns estimates to ground truth.

Physics

$$\mathrm{Drift}(\%)=100{\displaystyle \frac{\left|E\right(T)-E(0\left)\right|}{max\left(E\left(0\right),\epsilon \right)}},E\left(t\right)=T\left(t\right)+V\left(t\right).$$

(89)

$$\parallel g_{+}\left(q\right){\parallel }_{\infty }{\left(\mathrm{non}\text{-}\mathrm{penetration}\right),\parallel g\left(q\right)\parallel }_{\infty }\left(\mathrm{equality}\right),{\parallel J\left(q\right)v\parallel }_{\infty }\left(\mathrm{velocity}\text{-}\mathrm{level}\right).$$

(90)

Spatial Estimation

$$\mathrm{ATE}=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N\parallel {\mathbf{x}}_i-\left(S{\widehat{\mathbf{x}}}_i\right){\parallel }_2^2}.$$

(91)

$$R_i={\left(T_i^{-1}{T}_{i+\Delta }\right)}^{-1}\left({\widehat{T}}_i^{-1}{\widehat{T}}_{i+\Delta }\right),{\mathrm{RPE}}_{\mathrm{trans}}=\mathrm{RMSE}\left(\mathrm{trans}\left(R_i\right)\right),{\mathrm{RPE}}_{\mathrm{rot}}=\mathrm{RMSE}\left(\angle \left(R_i\right)\right).$$

(92)

$${\mathrm{RMSE}}_{\mathrm{repr}}=\sqrt{{\displaystyle \frac{1}{\left|\mathcal{P}\right|}}\sum _{p\in \mathcal{P}}\parallel {\mathbf{u}}_p-\pi \left(K{\widehat{T}}_p{\mathbf{X}}_p\right){\parallel }_2^2}.$$

(93)

Rendering and Systems

$$\mathrm{Missed}\text{-}\mathrm{frame}\mathrm{ratio}={\displaystyle \frac{\#\{t_{\mathrm{frame}}>\mathrm{budget}\}}{\#\left\{\mathrm{frames}\right\}}}.$$

(94)

Report MTP as median and 95th percentile (ms) from sensor timestamp to photons- on-panel.

Comfort

$$\mathrm{Time}\text{-}\mathrm{to}\text{-}\mathrm{discomfort}=min\{t\mid {\mathrm{SSQ}}_{\mathrm{Total}}\left(t\right)\ge \tau \}.$$

(95)

$$\mathrm{Dropout}\mathrm{rate}={\displaystyle \frac{\#\left\{\mathrm{participants}\mathrm{who}\mathrm{stop}\mathrm{early}\right\}}{\#\left\{\mathrm{participants}\right\}}}.$$

(96)

6.7.3. Reporting Template (Concise)

(i)

Task/dataset and total budget (per-eye resolution/refresh, frame budget).

(ii)

Metrics from Table 7, with window sizes and alignment choices.

(iii)

Foveation policy and salient system settings (e.g., reprojection, timewarp).

(iv)

Statistical summaries (median/p90/p99 with n and confidence intervals).

(v)

Ablations isolating learned vs. structure-preserving components.

6.8. Learning-Based Components: Practical Limitations Under Real-Time VR Constraints

6.8.1. Computational Overhead and Latency Variability

Learning-based components can reduce modeling burden or enable new capabilities, but they also introduce inference overhead and latency variability that are difficult to reconcile with strict VR frame budgets. Neural rendering and learned surrogates typically add extra stages (network evaluation, feature fetching, cache management) whose costs are not always amortized across frames, especially when scene content, view-dependent effects, or contact states change rapidly. Even when the median cost appears acceptable, occasional spikes caused by content-dependent workloads, thermal throttling, driver/runtime effects, or concurrent processes can trigger missed deadlines and visible judder. This is particularly problematic in stereo rendering, where small scheduling slips can accumulate into motion-to-photon latency increases and discomfort. Consequently, evaluation should emphasize tail behavior (e.g., percentile frame times) and include explicit fallback policies that preserve interaction continuity when learned components exceed budget.

6.8.2. Generalization and Out-of-Distribution Failures

A second limitation is robustness under distribution shift. Models trained on curated datasets can fail under novel lighting, motion patterns, occlusions, sensor noise, or uncommon user morphologies. In tracking and world-locking pipelines, such failures manifest as drift, sudden pose jumps, or loss of tracking, which directly undermines spatial stability and can produce strong discomfort. Similarly, learned avatar or body/hand models may degrade for users whose proportions or motion styles are underrepresented in training data, increasing retargeting error and reducing embodiment. Learned physics surrogates face comparable risks: contact-rich edge cases (multi-contact, high-velocity impacts, or frictional stick–slip transitions) can violate learned assumptions and yield non-physical behavior. For real-time VR deployment, these failure modes argue for hybrid designs in which learned outputs are continuously validated by geometric/physical consistency checks and are automatically reverted to classical estimators/solvers when confidence drops.

6.8.3. Deployment Guardrails and Recommended Practice

From a deployment perspective, the most reliable pattern is to treat learning as an augmentation layer rather than a single point of failure. Classical pipelines remain attractive because they provide predictable runtime, clearer debugging signals, and graceful degradation modes (reduced quality rather than catastrophic loss). Learning is most appropriate when its scope is well-bounded (e.g., accelerating a specific query, providing priors for optimization, or improving personalization) and when runtime monitors can detect anomalies early. Practically, systems should (i) profile performance under worst-case content and device states, (ii) reserve explicit margins for variability, (iii) implement fallback paths that maintain tracking stability and interaction safety, and (iv) report these guardrails as part of reproducibility. These considerations complement the broader limitations discussed later (Section 7.4) and motivate hybrid architectures that combine learned priors with geometric and physical constraints.

7. Conclusions

7.1. Synthesis: Toward Geometry-Aware, Perception-Aware, and Learning-Augmented VR

This survey has argued that the clearest path for real-time VR is a synthesis of three complementary lenses: geometry-aware models that preserve structure in dynamics and pose; perception-aware scheduling that allocates computation where it matters for users; and learning–augmented components that accelerate or calibrate core loops without breaking invariants. In practice, constraint-consistent updates and semi-implicit/symplectic stepping support physically plausible motion [2], perceptual tolerances gate adaptive effort across physics and rendering [16,17,22], and differentiable layers couple estimation and inverse problems to image formation and scene priors [10,13].

7.2. Domain-Specific Advances and Challenges

Physics. Progress stems from stable time stepping. Data-driven parameter identification and projection-based constraints further improve robustness. Nonsmooth frictional contact remains the main obstacle for differentiable pipelines and learned controllers [21]. Estimation. On-manifold optimization fused with differentiable rendering unifies tracking and calibration. This approach reduces drift and improves relocalization under fast motion [13,40]. Rendering. Hybrid rasterization–neural methods are pushing dual-eye 90/120 Hz. Representative techniques include radiance caching and factorized tiny networks. These maintain photometric fidelity [10,11,12]. Embodiment and Haptics. Implicit neural avatars with differentiable IK reduce retargeting error from sparse sensors [39,52,70]. Passivity-aware impedance control and reduced-order models sustain kHz haptics alongside visual refresh. Device/scene co-design and stability margins under latency remain open challenges [60,61,63]. Comfort-aware controllers that close the loop with cybersickness predictors are promising, but they need broader validation [66,67,68,71].

7.3. Methodological Recommendations

To make advances comparable and cumulative, we recommend a common reporting battery. For physics, include long-horizon energy drift, momentum/constraint violations, and contact stability traces [2]. For estimation, report ATE/RPE and per-frame reprojection RMSE with defined train/test splits and motion profiles [13,40]. For rendering/systems, provide frame-time percentiles (P50/P90/P99), motion-to-photon latency, and binocular synchronization, alongside image metrics; for interaction/comfort, include standardized cybersickness outcomes with pre-registered thresholds and time-to-discomfort analyses, plus ablations of perception-informed policies [16,17,66,67]. Where learning is used, ablate the surrogate’s role, characterize failure modes, and disclose memory/bandwidth profiles relevant to headsets.

7.4. Limitations and Future Outlook

Three cross-cutting limitations persist. First, memory/bandwidth ceilings constrain neural scene/shape representations in large or dynamic environments. Second, visibility- and shading-dependent gradients bias differentiable pipelines, complicating robust calibration and inverse problems [13]. Third, comfort models trained on narrow demographics or content may not generalize across tasks and devices [67,68,71,72,73,74,75,76].

Near-term opportunities include:

(i)

Structure-preserving, differentiable solvers for contact and friction that remain stable with learned surrogates [21].

(ii)

Tightly coupled on-manifold estimation with hybrid neural rendering for joint pose–layout inference at VR frame rates [11,12,40].

(iii)

Personalized embodiment and comfort-aware control that adapt in situ while honoring safety margins [52,66,70].

Carefully combining structure, perception, and learning offers a scalable route to stable and deeply immersive VR.