Enhancing Anatomy Teaching and Learning Through 3D and Mixed-Reality Tools in Medical Education ^†

Abstract

We developed an innovative system to enhance medical education, specifically in teaching anatomy and surgical techniques. We applied three-dimensional (3D) and mixed-reality (MR) technologies to create immersive learning experiences. The system enables students to gain hands-on experience in a virtual environment and understand anatomical structures and related surgical procedures. We designed the system architecture, user interface, and educational methodologies. By integrating modern technology into teaching methods, students can engage and improve learning efficiency, and the potential of 3D and MR technologies is validated in medical education.

1. Introduction

Lung anatomy represents challenges in thoracic surgery education, primarily due to its substantial anatomical variability and intricate three-dimensional architecture. Traditional teaching methods in thoracic surgery rely on anatomical textbooks and two-dimensional imaging materials for knowledge transmission. However, this conventional method fails to adequately convey the complex three-dimensional anatomical relationships of pulmonary structures. Moreover, the transition from open surgery to thoracoscopic procedures has complicated the learning curve, as surgeons must adapt to limited tissue manipulation and reduced tactile feedback [1].

To enhance the understanding of anatomical relationships and facilitate preoperative planning, three-dimensional (3D) visualization techniques are used. Advanced imaging technologies, including 3D computerized tomography (CT)-based reconstruction and simulation, are integrated into immersive extended reality (ER) platforms, such as virtual reality (VR), mixed reality (MR), and augmented reality (AR) [2,3]. These immersive technologies enable healthcare professionals to visualize, manipulate, and interact with patient-specific 3D models in fully virtual and hybrid simulated environments [4,5,6]. VR technology, in particular, bridges the gap between theoretical knowledge and practical training by allowing physicians to acquire cognitive understanding and technical skills in a risk-free environment outside the operating room.

In this study, we developed an innovative educational system to enhance thoracic surgical training. We examined the system architecture, user interaction modalities, pedagogical methodology, and learning outcomes of our VR-based platform. The system integrates interactive virtual environments into anatomical visualization, enabling medical trainees to acquire hands-on experience through immersive simulation. The system facilitates an understanding of complex pulmonary anatomical structures and associated surgical procedures, while providing a risk-free environment for skill development and procedural practice.

2. Background and Related Works

2.1. MR in Medical Education

Contemporary medical education is experiencing a paradigm shift driven by the proliferation of mobile MR technologies, encompassing smartphones, AR, VR, and 3D printing. These advanced technologies extend beyond traditional video-based educational methodologies and substantially enhance existing physical simulation training approaches. While conventional teaching models—including mannequin-based training, role-playing exercises, standardized patient interactions, and cadaveric dissection—remain valuable, digital media innovations have fostered diverse instructional modalities that significantly expand the scope and depth of clinical skills training [7,8].

Learners increasingly expect educational experiences characterized by interactive simulations and rich sensory engagement [9]. This expectation has transformed medical education from traditional didactic lectures toward self-directed learning approaches [10], online educational platforms [11], and experiential simulation-based training [12]. As empirical evidence continues to demonstrate the effectiveness of simulation and MR in enhancing learning outcomes [13] with the ubiquity of mobile devices [14], mobile MR tools integrated into medical education have become increasingly prevalent. In medicine, MR systems have been implemented in foundational subjects, particularly anatomy education. Zhang et al. [15] investigated the pedagogical advantages of direct manipulation of anatomical structures through stereoscopic 3D environments utilizing joystick controllers. Based on the results, Kucuk, Kapakin, and Goktas [16] examined the impact of AR-enhanced anatomy education on student learning effectiveness and cognitive load.

An advantage of MR lies in its capacity to facilitate real-time interactive integration between physical environments and virtual simulations. This distinctive capability has demonstrated remarkable potential in medical education and offered unprecedented opportunities for experiential learning and skill acquisition. As its applications expand, related research has progressed substantially, yielding innovative pedagogical approaches and transformative educational methodologies. The seamless integration of virtual and physical elements enables learners to engage with complex medical concepts and procedures in unprecedented ways through conventional teaching methods, thereby revolutionizing medical education paradigms and opening up new avenues for educational innovation.

2.2. Volumetric Video and 3D Capturing

Volumetric video technology has been advanced based on established technologies, primarily photogrammetry, a technique that enables three-dimensional object reconstruction from multiple two-dimensional images [17]. Unlike conventional surgical instructional videos that merely present specific techniques in limited surgeries, volumetric video technology enables surgical spatial visualization from the user’s perspective. This technology transcends the limitations of traditional videos and 360-degree stereoscopic recordings by using full-spectrum scene interaction, such as translational movements (anterior–posterior, superior–inferior, and lateral) and rotational movements (lateral tilt and anterior–posterior tilt). Through precise user position tracking, the native VR environment facilitates natural navigation in a virtual surgical environment, delivering an enhanced immersive experience.

This technological innovation aligns with contemporary trends in medical education and training. Simulation technologies have been extensively implemented in healthcare education in the civilian and military medical sectors [18], encompassing medical mannequins for psychomotor skill development and virtual training systems for procedural knowledge acquisition. Extensive research results have validated the effectiveness of simulation-based education, garnering widespread recognition within the medical community [19,20,21]. While less prevalent than physical simulation devices, games and virtual simulations demonstrate substantial advantages in learner engagement and educational outcomes [22,23]. Although related research continues to evolve, AR and VR technologies exhibit remarkable potential for enhancing learning experiences in healthcare education [24,25].

In this context, we investigated the technical workflow of volumetric video, encompassing the acquisition, generation, and semantic labeling of hybrid animation, and developed an enhanced educational system for thoracic surgery through the integration of volumetric video technology and wearable MR devices.

3. System Design and Implementation

3.1. Hardware and Software

In this study, we implemented Holocode https://support.makeblock.com/hc/en-us/articles/20072180888343-About-Halocode) (accessed on 8 August 2025) as the primary data processing and control unit of the system. Holocode processes input signals from compatible sensors, including angular and magnetic sensors, to facilitate user data tracking, gesture recognition, and real-time interaction control. With the assessment component, Holocode presents the question-answering interface and coordinates the logical processing of response selection, ensuring fluid interaction throughout the educational experience.

For the MR visualization component, we employed Microsoft HoloLens 2 to render volumetric videos in the system (Figure 1). The implementation system comprises MR Toolkit 2.7 and Unity 2019.4.23. Beyond conventional surgical skills training, the system incorporates an interactive assessment module. The evaluation interface adopts angular and magnetic sensors for response selection, enabling learners to indicate their choices through sensor-based interactions. The system incorporates automatic response logging functionality and delivers instantaneous feedback alongside comprehensive evaluation based on predetermined assessment criteria.

Figure 1. Devices of developed system: Holocode (left), Azure Kinect DK (middle), Microsoft HoloLens 2 (right).

3.2. System Architecture

Surgical skill acquisition without expert supervision yields suboptimal outcomes. To enhance the effectiveness of surgical training, the developed system incorporates multiple high-precision sensors for comprehensive operative data tracking and recording. Inertial and magnetic sensors were used to collect operational parameters, including spatial positioning of surgical instruments, movement trajectories, and procedural angles. These high-fidelity data metrics were used to evaluate procedural standardization and provide quantitative benchmarks for surgical skill progression. The systematic collection and analysis of these objective parameters of the system for continuous performance assessment and skill refinement in surgical training are shown in Figure 2.

Figure 2. System architecture.

3.3. Interactive Design

A distinguishing feature of the developed system is its capacity to facilitate physical movement replication, gesture memorization, and spatial comprehension of surgical instruments and anatomical structures before operation. This enables it to overcome the limitations of traditional fixed-screen learning approaches. This design was developed to effectively bridge the gap between simulated training and clinical practice in instrument positioning, equipment familiarization, and workflow optimization. The system transforms the educational experience from passive observation to active participation, from 2D screen-based learning to immersive spatial interaction.

The system comprises two distinct phases. In the initial volumetric video learning, learners engage with surgical content through HoloLens 2, with free manipulation. Subsequently, the system leads to interactive assessment, where learners utilize angular and magnetic sensors for response selection. The system provides real-time feedback on responses and generates comprehensive performance reports upon completion (Figure 3).

Figure 3. Volumetric video surgery instruction.

4. Conclusions

The convergence of MR and volumetric video technologies enhances thoracic surgical education and training. The system developed integrates wearable MR devices into volumetric video technology, complemented by multi-sensor data acquisition, to create an advanced educational platform for thoracic surgery training. The system delivers immersive surgical instruction while incorporating quantitative assessment mechanisms through an interactive evaluation module, providing objective metrics for educational efficacy evaluation. The system’s dual functionality, immersive visualization, and performance analytics enable surgical skill acquisition and assessment. Positive user feedback suggests potential for expanded implementation and validation.