A Data-Driven XR Environment for Understanding Probe Manipulation in Musculoskeletal Ultrasound

Abstract

Competency in musculoskeletal (MSK) ultrasound requires learners to relate probe manipulation to spatial reasoning, image projection, and the appearance of characteristic artefacts, which remains challenging during early training due to the limited spatial context provided by conventional instructional resources. This study investigates whether reconstructing real MSK ultrasound examinations in an immersive extended reality (XR) environment is perceived as useful for early familiarisation with probe handling and image interpretation. The proposed system reproduces ultrasound acquisitions using synchronised ultrasound video, six-degree-of-freedom probe tracking, and surface scans acquired from cadaveric specimens, enabling the reconstruction of spatially accurate probe trajectories with each ultrasound frame linked to a corresponding position and orientation. Within the XR environment, users can interactively explore these trajectories or observe automated playback in which the recorded probe motion is presented together with the corresponding ultrasound sequence. An exploratory evaluation with healthcare professionals was conducted to assess perceived usefulness and clarity of spatial relationships. The results indicate that participants perceived spatially coherent playback of real ultrasound examinations in XR as a potentially useful aid for understanding probe–image relationships. These findings suggest the feasibility of this approach as a complementary resource for introductory MSK ultrasound training.

1. Introduction

Ultrasound imaging is widely used in clinical practice due to its real-time feedback, portability, absence of ionizing radiation, and relatively low cost [1,2]. However, the acquisition and interpretation of ultrasound images remain technically demanding. Unlike tomographic techniques such as computed tomography or magnetic resonance imaging, ultrasound does not provide direct anatomical cross-sections. Instead, image formation results from complex acoustic interactions between tissues, making image appearance highly dependent on acquisition parameters and operator actions. From a technical perspective, ultrasound therefore constitutes a tightly coupled perception–action process, in which image interpretation cannot be separated from probe manipulation [3].

Effective ultrasound training requires the simultaneous acquisition of spatial reasoning, motor coordination, and perceptual interpretation skills [4]. Small variations in probe position, orientation, or contact can lead to substantial changes in image appearance, contributing to a steep learning curve, particularly for novice users. These challenges are further amplified in musculoskeletal (MSK) ultrasound, where tissues such as muscles, tendons, and peripheral nerves exhibit pronounced anisotropic behavior. Accurate visualization of muscle fibers, nerve fascicles, and subtle pathological features demands precise probe alignment and a clear understanding of three-dimensional anatomical relationships, placing high demands on probe handling skills and spatial awareness.

In this context, extended reality (XR) technologies, encompassing virtual reality (VR) and augmented reality (AR), offer promising opportunities for ultrasound education and training [5]. Immersive environments enable spatially coherent visualization and embodied interaction, allowing probe motion, anatomy, and image feedback to be experienced within a unified three-dimensional context. This work presents a XR-based framework for musculoskeletal ultrasound training that emphasizes probe manipulation and spatial interpretation by integrating real probe motion capture, anatomical digitization, and synchronized ultrasound data. The results indicate that participants perceived the system as useful for the exploration and analysis of probe–image relationships, suggesting the potential relevance of XR-based approaches in this domain.

2. Related Work

Musculoskeletal ultrasound is widely recognized as a highly operator-dependent modality requiring coordinated probe manipulation, spatial reasoning, and image interpretation. More broadly, ultrasound learning is frequently reported as challenging for beginners due to the need to simultaneously understand ultrasound physics, sectional anatomy, and the relationship between probe movements and the resulting two-dimensional image. Prior educational research has shown that novices particularly struggle with determining where the probe should be placed, angled, and rotated to obtain specific anatomical sections [6]. In addition, a recent systematic review of musculoskeletal ultrasound learning methodologies highlighted the considerable heterogeneity of existing training approaches across specialties and institutions, as well as the lack of high-quality comparative studies establishing an optimal pedagogical model [7]. This variability in educational design may further contribute to inconsistencies in skill acquisition and reinforces the intrinsic complexity of MSK ultrasound training. This difficulty in bridging theory and hands-on probe positioning underscores the persistent gap between conceptual knowledge and practical skill acquisition.

To address the steep learning curve and the limited availability of supervised practice, a variety of technology-enhanced ultrasound training systems have been described in the literature. These systems differ in their degree of physical realism, spatial awareness, and the extent to which probe pose is coupled with ultrasound image formation. The following subsections review representative approaches, grouped according to the type of training environment and the underlying simulation methodology.

2.1. Phantom-Based and Computer Simulators

Ultrasound simulators have been developed to provide structured and repeatable training environments without requiring access to patients or cadaveric material. Early systems primarily relied on screen-based platforms in which users manipulated a virtual probe using a mouse, keyboard, or simplified input device. These systems typically generated synthetic ultrasound images from virtual anatomical models or pre-recorded datasets. Their main advantages lie in scalability [8], accessibility, and the possibility of integrating automated assessment and predefined training scenarios.

Commercial computer-based simulators typically couple a tracked mock probe with either physical phantoms or virtual anatomy and generate ultrasound images from pre-recorded 3D volumes or physics-based rendering [9]. Such systems enable repeated practice, case databases, and automated feedback. They have shown positive learning effects, including the accelerated acquisition of fetal echocardiography standard planes and performance comparable to that of experts after structured simulation-based training [10]. However, the review by Blum et al. [9] emphasizes that rigorous quantitative evaluations are rare, and long-term transfer to clinical performance remains under-documented.

Hybrid simulators integrate realistic phantoms and software to explicitly visualize the 3D relationship between probe, scan plane, and internal anatomy. Freschi et al. [11] showed that a mixed-reality phantom system significantly improved novices’ ability to visualize a target structure compared with a physical simulator (78% vs. 45%) and was rated highly for 3D perception, underscoring the educational value of exposing spatial probe–image relationships beyond conventional 2D screens. Still, these systems rely on synthetic or CT-derived anatomy [12] rather than replaying real examinations.

Computer-based ultrasound simulators generate synthetic images by modeling, to varying degrees of approximation, the physical behavior of an ultrasound system. These frameworks typically incorporate parameters such as transducer position and orientation, acoustic properties of tissues, and probe–tissue interaction. Synthetic image formation may rely on different types of input data, including pre-recorded ultrasound volumes, cross-sectional imaging datasets (e.g., CT or MRI), or fully constructed three-dimensional anatomical models. For example, Starkov et al. [13] combine path-tracing techniques with animated anatomical models to dynamically integrate simulated and pre-acquired ultrasound data, thereby increasing visual realism. From a methodological perspective, existing simulators can broadly be classified according to their underlying image-generation strategy, including physics-based acoustic wave propagation models [14,15], geometric or ray-tracing approaches [16,17], and more recent data-driven or artificial intelligence-based techniques [18].

Although many of these approaches are suitable for interactive applications, they often lack the fine-grained echotexture required for musculoskeletal ultrasound training, where subtle variations in the appearance of nerves, fascial layers, and intramuscular fat are diagnostically relevant. Moreover, many existing systems primarily focus on simulating acoustic propagation phenomena rather than supporting early-stage learning aimed at understanding probe orientation, spatial alignment, and the relationship between probe pose and the resulting image.

2.2. VR Simulators and Immersive Experiences

Recent developments have increasingly incorporated immersive technologies, particularly virtual reality and augmented reality, to enhance spatial understanding and interaction. In VR-based simulators, users manipulate tracked controllers or probe replicas within a three-dimensional virtual environment, where anatomical models and simulated ultrasound images are co-visualized. UltRASim uses a Head Mounted Display (HMD), virtual patient, and probe/needle manipulation (with or without haptics) to train spatial hand–eye coordination [19]. Similarly, LiVRSono presents an immersive VR training system for intraoperative ultrasound that integrates simulated ultrasound, patient-specific data, and haptic feedback to facilitate the development of a spatial mental model linking probe position and image appearance [20]. In this system, the immersive setup aims to emulate the clinical configuration, including probe handling and monitor placement, thereby reinforcing visuospatial skills in a realistic surgical context.

Augmented and mixed reality approaches have also been proposed to position ultrasound data directly within the user’s visual field. HoloPOCUS, for instance, introduces a mixed-reality ultrasound framework that overlays tracked two-dimensional slices and reconstructed three-dimensional volumes onto the patient using a head-mounted display [21]. By combining stereo fiducial tracking with real-time reprojection, the system aims to reduce cognitive load associated with mentally registering ultrasound images to patient anatomy and has demonstrated improvements in task performance in phantom-based studies. More generally, AR head-mounted displays that overlay live ultrasound images at the physical probe position have been reported to improve procedural accuracy and perceived spatial understanding, although ergonomic and usability challenges remain [22].

Despite these advances, several limitations remain, particularly in musculoskeletal ultrasound training. Many VR and AR simulators rely on simplified anatomical models and approximate acoustic rendering techniques. While such approaches are sufficient for illustrating general principles of image formation, they often lack the fine-grained echogenic detail required for MSK interpretation. Additionally, although immersive technologies improve spatial visualization, the linkage between probe manipulation and realistic image output may remain constrained by the underlying simulation models.

2.3. Tracking, Registration, and Spatially Aware Systems

Several systems incorporate optical, electromagnetic, or mechanical tracking technologies to record probe trajectories and, in some cases, to register probe pose to preoperative imaging or three-dimensional anatomical models. These platforms can visualize the ultrasound plane in space and support applications such as navigation guidance or skill assessment. A more explicit approach to spatial guidance is presented by Nishi et al. [23], who used a motion-capture interface that displays real-time probe position and angle compared to an expert reference during kidney ultrasound. Beginners could reproduce the expert pose within a few millimeters and degrees, achieving very similar cross-sections without prior conventional probe training, suggesting that visualized pose matching is sufficient to teach reproducible probe positioning in simple scenarios. However, this approach focuses on reaching a single static target configuration rather than understanding continuous trajectories or complex musculoskeletal scanning patterns.

While tracking enhances spatial awareness, many of these systems have been developed primarily for interventional guidance rather than structured training. Moreover, a substantial portion of the literature focuses on abdominal, obstetric, or cardiac ultrasound. In the musculoskeletal domain, explicit synchronization between a real ultrasound frame and its exact 6DOF probe pose, captured in a reusable educational dataset, remains comparatively limited.

2.4. Identified Gap

Survey data from accredited sonography programs show that simulation is widely used and especially valued for improving transducer manipulation [24]. However, many existing VR/AR systems either do not support full 6DOF interaction with realistic ultrasound responses [25], rely on synthetic anatomy instead of real recorded exams [11,19,26], or only provide discrete pose “targets” without continuous spatial exploration of expert maneuvers [23]. There is thus a clear gap for systems that let learners step into a spatial replay of real ultrasound sessions, inspect expert probe paths in 3D, and actively “scrub” through space and time using recorded video and tracking data, potentially supporting a deeper understanding of how continuous probe motion generates changing image content. The framework presented in this work combines real cadaveric MSK ultrasound examinations with synchronized 6DOF motion capture and immersive visualization, aiming to integrate anatomical realism, spatial awareness, and reusable educational content.

3. Materials and Methods

3.1. Data Acquisition

The data acquisition protocol was designed to capture, with high temporal accuracy, the relationship between ultrasound probe motion and the resulting image output, with the objective of enabling subsequent integration into an immersive XR environment. Optical motion capture was combined with synchronised ultrasound video recording in a controlled laboratory setting.

Probe motion was recorded using an optical motion capture system composed of five OptiTrack Flex 3 cameras (NaturalPoint Inc, Corvallis, OR, USA) [27], operating at a spatial resolution of 640 × 480 pixels and a capture frequency of 100 Hz. Four cameras were arranged in two opposing lateral pairs surrounding the acquisition workspace, providing overlapping views of the probe from both sides of the capture volume. A fifth camera was positioned above the workspace to provide a zenithal view. This configuration was selected to ensure adequate coverage of the probe workspace and to reduce occlusions during scanning. Figure 1 provides a schematic representation of this multi-camera arrangement, showing the lateral camera pairs positioned on opposing sides of the workspace and the zenithal camera, with the cadaveric specimen positioned centrally on the gurney.

Figure 1. Schematic layout of the optical motion capture system used to record probe motion, showing four lateral cameras arranged in opposing pairs and a fifth zenithal camera above the acquisition workspace.

The system calibration was performed above the gurney following standard OptiTrack calibration procedures and prior to positioning the cadaveric specimen. The calibration was executed in the absence of the cadaver to prevent interference with marker visibility and to ensure the stability of the global reference frame. Following the completion of the calibration procedure, the cadaver was carefully positioned onto the designated gurney and remained in a fixed position throughout data acquisition. The position and orientation of the probe were tracked and recorded using Motive motion capture software (version 2.1.1) [28].

Concurrently, a surface scan of the cadaveric specimen was obtained using an Apple iPad Pro tablet (Apple, Cupertino, CA, USA) equipped with a LiDAR sensor. The surface digitization was performed using the same transform origin defined during the motion capture calibration, ensuring spatial consistency between the captured probe motion and the reconstructed anatomical surface within the virtual environment.

The ultrasound image data acquired during the scanning procedure was captured using a linear HFL50 probe. The video was recorded using a resolution of $1920\times 1080$ pixels and a frame rate of 10 frames per second. This rate was selected as a practical balance between temporal resolution and storage volume, and is consistent with the slow, controlled probe movements characteristic of MSK ultrasound scanning. Figure 2 shows the complete experimental arrangement, including the optical motion capture cameras mounted on overhead rails, the cadaveric specimen on the gurney, and the ultrasound equipment used during synchronized data acquisition.

Figure 2. Photograph of the experimental setup used for ultrasound acquisition and probe tracking with the optical motion capture system.

To temporally align probe motion and ultrasound imagery, an ad hoc Python (version 3.12) utility was developed. The application receives a real-time stream of motion capture data from Motive via a UDP socket and synchronizes this data with the corresponding ultrasound video frames. The synchronization process yields two outputs: an MP4 file containing the ultrasound video and a JSON file storing the synchronized motion data. The JSON structure, illustrated in Listing A1 (Appendix A), consists of an initial calibration segment, followed by frame-level entries comprising the probe identifier, a timestamp relative to the initiation of the recording, and the associated three-dimensional position and quaternion-based orientation. This representation enables precise replay and analysis of probe motion in conjunction with ultrasound imagery within the virtual environment.

As illustrated in Figure 3, the system architecture diagram depicts the data flow between the motion capture system, the ultrasound video capture module, the synchronization utility, and the generated video and JSON files. This architecture constitutes the foundation for the XR-based visualization and interaction pipeline that will be delineated in the subsequent sections.

Figure 3. System architecture illustrating the data flow between the motion capture system, ultrasound video capture, synchronization process, and the resulting synchronized datasets.

3.2. Integration into the Virtual Reality Environment

The integration of the acquired data into an immersive environment was implemented using the Unity game engine (version 6000.0.61f1) in combination with the OpenXR API. This combination enabled deployment across a range of VR devices while maintaining a unified interaction and rendering pipeline. Unity was selected due to its support for real-time rendering, VR interaction, and efficient handling of heterogeneous data sources, including surface meshes, time-dependent motion data, and video textures.

The surface mesh obtained from the LiDAR scan was imported into Unity and placed within the virtual scene using the same reference transform defined during the motion capture calibration. This ensured spatial consistency between the anatomical surface and the recorded probe motion (see Figure 4). To express all probe positions and orientations within this coordinate frame, a transformation was applied to convert the recorded motion data from the right-handed coordinate convention of the OptiTrack system to the left-handed convention adopted by Unity. Each resulting pose was then composed with the inverse calibration transform derived from the calibration poses stored in the JSON file, placing probe trajectories and ultrasound visualisations in spatial alignment with the cadaveric anatomy and preserving the geometric relationships observed during data acquisition.

Figure 4. 3D surface mesh of the cadaveric specimen reconstructed from the LiDAR scan and positioned within the Unity scene.

The motion capture data stored in the JSON file were processed in an offline preprocessing step performed once per acquisition. Because the motion capture system and the ultrasound video operated at different frame rates, the probe trajectory was temporally resampled to align the motion data with the ultrasound video frames. Position was resampled using linear interpolation, while orientation was resampled using spherical linear interpolation (SLERP) to ensure smooth and geometrically consistent transitions between recorded quaternion poses. This procedure produces a synchronized dataset in which each ultrasound frame is associated with a corresponding probe position and orientation. The resulting dataset can subsequently be reused across different applications without requiring the preprocessing stage to be repeated.

A set of keyframes was extracted from the processed motion data, and these were then used to reconstruct the recorded ultrasound scanning trajectories within the virtual environment. The reconstructed trajectories were represented as scan paths, which were superimposed onto the anatomical surface mesh (see Figure 5). These pathways depict probe movement spatially and function as a visual aid for analysing scanning strategies and anatomical coverage. Each generated path incorporates a set of colliders, thereby facilitating interaction with the XR system. These colliders are utilized to ascertain when the user’s controller, whether a standard VR controller or a haptic input device, positions the virtual probe over a recorded scan location. In scenarios where multiple scan paths are available, the user is required to make an explicit selection of the active path. This selection enables interaction and feedback for that specific recording.

Figure 5. Reconstructed probe trajectory from motion capture data. Top: extracted keyframes defining the scan path. Bottom: colliders associated with the trajectory to support interaction in the virtual environment.

The ultrasound video data were loaded alongside the motion data, and internal data structures maintain the temporal relationship between video frames and their corresponding probe poses. Video frames were stored as an array of textures uploaded to the GPU, enabling low-latency random access to individual frames without playback stuttering. The ultrasound video is displayed on an interactive, grabbable panel within the virtual environment (Figure 6). However, video playback is driven directly by the spatial interaction between the virtual probe and the reconstructed scan paths. By manipulating the virtual probe along the recorded trajectory, users can perform a variety of actions, including scrubbing through the ultrasound sequence in a non-linear manner, jumping between positions, or traversing the path in reverse. This enables direct linkage between probe motion and image playback.

Figure 6. Interactive panel displaying the ultrasound video within the virtual environment. Video playback is controlled by the probe position along the reconstructed scan path.

3.3. Dataset

The dataset employed in this study was obtained from a single cadaveric specimen corresponding to a 65-year-old male and was specifically designed to support the exploration of musculoskeletal ultrasound training within an immersive XR environment. The acquisition of data was centered on two anatomical regions of particular interest: the left upper limb and the left lower limb. For each region, three ultrasound scan paths were recorded, corresponding to the internal, anterior, and external faces, resulting in a total of six recorded scanning paths.

All ultrasound recordings were performed by a sonographer with extensive experience in musculoskeletal ultrasound. This approach ensured that probe placement, orientation, and scanning trajectories adhered to established clinical protocols and were oriented towards the visualization of anatomically relevant musculoskeletal structures. For each anatomical region, probe motion was guided by the orientations and movements commonly used to identify significant tissue features, such as muscle architecture, connective tissues, and peripheral nerves.

The duration of each scan path is approximately 40 s. The ultrasound video was recorded at a spatial resolution of $1920\times 1080$ pixels and a temporal resolution of 10 frames per second. The corresponding probe motion data capture both position and orientation throughout the scanning process, enabling the reconstruction of clinically meaningful trajectories within the virtual environment. In order to conduct this experiment, the LiDAR-derived surface mesh of the cadaveric specimen was first cleaned and then optimized prior to integration. This process resulted in a model composed of approximately 800,000 vertices with high-quality textures. This surface representation provides a detailed anatomical context for the visualization of probe motion and scan paths within the virtual environment.

4. Results

4.1. Visualization of Ultrasound Scan Paths

Within the virtual environment, the recorded ultrasound scan paths are presented as spatial overlays on the anatomical surface, thereby providing an immediate visual reference of how the expert scans were performed.

At the start of each session, once the user attaches the head-mounted display, the system prompts the selection of the interaction modality, allowing the user to choose between a standard VR controller or a haptic input device. The virtual environment then initialises according to the selected mode. In the present study, Meta Quest 3 devices were utilized, and the application was implemented in an extended reality (XR) configuration. This see-through configuration facilitates the visualization of virtual content within the user’s immediate physical environment. The selection of an XR mode was driven by training scenarios in which maintaining visual contact with instructors or other users might prove advantageous. Nevertheless, the application can also be operated in a fully immersive virtual reality mode when interaction with the physical environment is not required.

Once the environment is active, users can move freely around the scanned cadaveric specimen. The available ultrasound scan paths are displayed as highlighted trajectories over the anatomical surface, clearly indicating the regions where probe interaction with recorded scan data is available. These visual cues facilitate the identification and selection of specific scan paths by users, as well as the correlation of probe motion with external anatomical landmarks. The system enables the inspection of multiple paths corresponding to different anatomical faces, with the option to select the desired trajectory for individual inspection.

In addition to the scan path visualization, users are presented with a virtual panel displaying the ultrasound video. Users can reposition the panel freely within the virtual space to suit their viewing preferences.

4.2. Interaction Systems

The extended reality application offers a variety of interaction systems intended to address distinct exploratory and review scenarios within the context of musculoskeletal ultrasound training. Each interaction modality emphasizes a distinct aspect of the scanning process, ranging from unconstrained exploratory interaction to physically grounded probe manipulation and structured observation of expert scans. Collectively, these methods offer a range of perspectives on probe handling and ultrasound image interpretation.

4.2.1. XR-Based Free Exploration

In the XR-based free exploration mode, users interact with the system using a standard VR controller, to which a generic virtual probe is rigidly attached. This configuration enables probe motion to be directly driven by the user’s hand movements, thereby providing full freedom of motion in six degrees of freedom. Users are permitted to navigate freely around the scanned cadaveric specimen and explore the recorded scanning regions from arbitrary viewpoints (Figure 7).

Figure 7. Free exploration mode where users manipulate a virtual probe to interact with reconstructed scan paths and display the corresponding ultrasound frames.

As the user approaches a recorded ultrasound scan path with the virtual probe, the system provides multimodal guidance cues. In addition to a subtle vibrotactile response delivered through the VR controller, the currently active scan path undergoes a slight color change, visually highlighting the trajectory under interaction. This visual indicator serves to communicate that the probe is within a valid interaction region and that the corresponding scan path has been selected. The selection of paths is managed through colliders generated during the preprocessing stage. These colliders are responsible for detecting the spatial overlap between the controller and the recorded trajectory.

After a scan path has been selected, the internal data frame structure linked to that particular path becomes active. Once a scan path has been selected, the ultrasound video corresponding to that scan is rendered on the grabbable video panel. The playback of video is driven by the spatial relationship between the virtual probe and the scan path. As the user moves the probe along the trajectory, the shader renders the corresponding frame from the preloaded array of GPU-resident textures, updating the displayed ultrasound image in real time. This mechanism supports non-linear exploration of the scan, allowing users to move forward or backward along the trajectory without relying on conventional playback controls.

The XR-based free exploration mode utilizes a combination of visual highlighting, vibrotactile feedback, and spatially driven video rendering to reinforce the link between probe placement and ultrasound image formation. This mode maintains a responsive and intuitive interaction experience within the immersive environment.

4.2.2. Haptic Interaction

In the haptic interaction mode, probe manipulation is performed using a 3D Systems Touch device, thereby enabling physically grounded interaction through force feedback. This interaction modality relates to the phantom-based training approaches commonly reported in the literature, where physical devices or mock probes are used to reproduce the tactile component of ultrasound probe manipulation. In contradistinction to the XR-based free exploration mode, this interaction paradigm necessitates an explicit calibration step to ensure correct spatial alignment between the physical workspace of the haptic device and its virtual representation within the immersive environment.

Prior to interaction, a calibration procedure is performed to register the haptic device workspace with the virtual scene. This calibration is performed through the use of hand tracking, which involves the placement of a set of virtual reference spheres on the XZ plane. The user is instructed to physically touch these reference points with the haptic device end effector, thereby enabling the system to estimate the transformation that aligns the physical device workspace with the corresponding virtual coordinates. The calibration process is illustrated in Figure 8 (left). A quantitative characterisation of calibration accuracy, including registration error across trials and users, was not performed in this study and remains a direction for future technical validation.

Figure 8. Haptic interaction workflow. Left: calibration of the haptic device workspace using virtual reference spheres. Right: probe manipulation with force feedback using the 3D Systems Touch device within the immersive environment.

Following the completion of the calibration process, the virtual environment is initialized with a dedicated three-dimensional ultrasound probe model affixed to the haptic device. The physical workspace and kinematics of the device impose constraints on probe movement, resulting in a more limited but stable range of motion compared to that of the VR controller. During interaction, the haptic device provides force feedback when the virtual probe comes into contact with the anatomical surface, simulating probe–skin contact and reinforcing the perception of physical interaction. An illustration of the interaction employing the haptic device is presented in Figure 8 (right).

As in the XR-based mode, interaction with recorded scan paths is detected using the precomputed colliders associated with each trajectory. Upon the intersection of the virtual probe with a scan path, the corresponding ultrasound data become active and are displayed on the video panel. The integration of spatial calibration, constrained probe motion, and force feedback facilitates a targeted interaction modality that accentuates contact awareness and precise probe positioning within the recorded scan region.

4.2.3. Automated Playback Mode

The automated playback mode is independent of the selected interaction system and does not require active probe manipulation. In this mode, the system reproduces the recorded scan by animating the virtual probe according to the captured position and orientation data over time. The corresponding ultrasound frames are displayed synchronously, allowing users to observe the scan as it was originally performed.

This mode is intended primarily for detailed review and analysis. By visualizing the exact probe pose associated with each ultrasound frame, the system affords the opportunity to observe the relationship between probe positioning and characteristic imaging phenomena, such as bone shadowing or anisotropy effects commonly observed in musculoskeletal ultrasound. Automated playback enables careful inspection of expert scanning techniques and serves as a reference for comparison with interactive exploration modes.

4.3. Evaluation

The evaluation of the system was centered on assessing its usability and perceived suitability for musculoskeletal ultrasound training. To this end, two complementary studies were conducted using the System Usability Scale (SUS) [29], a widely adopted and technology-independent questionnaire for the evaluation of interactive systems. SUS offers a reliable measure of perceived usability and has been demonstrated to be suitable for small sample sizes and early-stage prototypes, a finding with particular relevance in the context of medical VR/XR applications.

The initial study comprised five expert participants who possess extensive experience in healthcare, anatomy, and ultrasound. The limited sample size is indicative of the inherent challenge in accessing highly specialized clinical and anatomical personnel, a constraint that is frequently documented in exploratory evaluations of medical training systems. Prior to the evaluation, experts received a brief, one-minute guided introduction to the use of the XR headset and controller to facilitate their familiarization with the immersive environment. The intention behind the concise nature of this introduction was to mitigate the occurrence of training effects that were not directly related to the system itself. Participants were then allotted five minutes to freely explore the virtual environment, interact with the ultrasound scan paths, and observe the corresponding ultrasound imagery. Figure 9 presents a photograph of the experts conducting the evaluation. Subsequent to the interaction session, participants completed the SUS questionnaire. The responses to each SUS item are documented in Table 1, while the aggregate SUS score for the expert group is outlined in Table 2.

Figure 9. Medical experts interacting with the XR-based ultrasound exploration system during the evaluation session.

Table 1. SUS questionnaire results for users with medical background.

User	SUS-1	SUS-2	SUS-3	SUS-4	SUS-5	SUS-6	SUS-7	SUS-8	SUS-9	SUS-10
1	5	2	4	5	5	2	4	1	5	1
2	5	1	5	4	5	1	4	1	4	2
3	5	5	5	3	5	5	5	5	5	3
4	5	1	5	1	5	1	5	1	5	1
5	5	1	4	2	5	1	5	2	5	2

Table 2. SUS score for medical background users.

User	SUS Score
1	80
2	85
3	60
4	100
5	90

A second usability study was conducted with five generalist users lacking a professional background in ultrasound or anatomy. The objective of this study was to assess the usability of the system for a broader audience and to identify potential barriers to interaction beyond expert use. The evaluation protocol was consistent with that of the expert study, comprising a concise introduction to the XR hardware, followed by an unrestricted exploration period within the environment. Subsequent to the completion of the interaction session, generalist users were asked to complete the SUS questionnaire. The distribution of individual SUS scores for this group is documented in Table 3, while the aggregate SUS score is presented in Table 4.

Table 3. SUS questionnaire results for users with no medical background.

User	SUS-1	SUS-2	SUS-3	SUS-4	SUS-5	SUS-6	SUS-7	SUS-8	SUS-9	SUS-10
6	5	2	5	1	5	1	5	2	5	2
7	5	1	4	2	4	1	5	1	5	2
8	5	3	5	2	5	2	4	1	5	1
9	5	1	5	3	5	1	5	1	5	1
10	4	5	4	4	5	3	5	1	5	2

Table 4. SUS score for users with no medical background.

User	SUS Score
6	92.5
7	90
8	87.5
9	95
10	70

Together, these two studies provide complementary perspectives on the usability of the proposed system, combining feedback from domain experts with assessments from non-specialist users. The mean SUS score obtained from generalist users was 87.0, while the expert group with a medical background reported a mean score of 83.0, resulting in an overall average SUS score of 85.0 across participants. According to established SUS scale, scores above 85 are considered to indicate excellent usability and high user acceptance [30]. These results therefore suggest that the system achieves a high level of perceived usability across both expert and non-expert populations, supporting its acceptability as an immersive tool within a musculoskeletal ultrasound training context.

5. Discussion

This study introduced an immersive XR-based framework for musculoskeletal ultrasound training that integrates synchronized probe motion, anatomical surface reconstruction, and real ultrasound imagery within a unified spatial environment. The results indicate that the system achieves high perceived usability across both expert and non-expert users, suggesting that an immersive XR approach of this kind is acceptable to the target population and warrants further investigation in this domain. From the perspective of current ultrasound education practices, these findings are consistent with the hypothesis that spatially grounded visualisation may be perceived as a useful resource for addressing some of the core challenges associated with learning probe manipulation, though empirical assessment of learning outcomes remains a necessary direction for future work.

Ultrasound training has traditionally relied on supervised bedside practice and on simulation-based systems that either use physical phantoms or purely virtual models. While these approaches provide valuable learning opportunities, they often separate anatomical context, probe handling, and image interpretation. In contrast, the proposed framework explicitly links real ultrasound images, familiar to clinical practitioners, with the precise position and orientation of the probe at the moment of acquisition. This spatial and temporal coupling affords the opportunity to observe how specific probe manipulations generate characteristic image phenomena, including shadowing effects, anisotropy, and projection-related changes. For early-stage learners, this direct correspondence is intended to support the development of an initial mental model connecting three-dimensional anatomy, probe motion, and two-dimensional ultrasound output.

The framework also affords opportunities for more autonomous learning. By situating training within a virtual environment, students can explore recorded scans without requiring continuous instructor supervision. The ability to replay expert trajectories, to interactively traverse scan paths in a non-linear manner, and to visualize probe pose in relation to the anatomical surface provides learners with opportunities for repeated, self-paced practice. Such autonomy may be particularly relevant in educational contexts where access to expert instructors is limited, and the self-contained nature of the application means it can be used independently outside formal teaching settings, including remote or home-based study.

The three interaction modalities serve distinct practical purposes and map onto different stages of a typical ultrasound training curriculum. Automated playback requires no active probe manipulation and is well-suited as a preparatory tool in introductory contexts, allowing students to observe expert probe trajectories and their corresponding image output before handling a physical probe. Free exploration provides unrestricted 6DOF movement without peripheral calibration, making it appropriate for learners who have acquired basic spatial orientation and are ready to develop an intuitive understanding of continuous probe-image correspondence through self-directed interaction. The haptic mode, with its calibration step, constrained workspace, and force feedback at the probe-surface interface, addresses a more focused stage of training in which learners practise precise probe contact and positioning. Beyond individual use, the system could be integrated into existing ultrasound courses as a supplementary resource between supervised clinical sessions, providing opportunities for independent review without requiring instructor availability.

The rendering performance of the system was assessed across two widely adopted XR devices, the Meta Quest 3 and Meta Quest Pro, both operating at refresh rates of 60 to 72 Hz. Throughout the evaluation sessions, no perceptible latency, frame drops, or visual artefacts were observed across any of the three interaction modalities on either device, suggesting that the spatially driven video rendering pipeline operates compatibly with real-time interaction requirements under the conditions tested.

This work should not be interpreted solely as the presentation of a standalone XR application. Rather, it introduces a framework that combines motion capture, anatomical digitization, synchronized ultrasound recording, and immersive visualization. The data acquisition and synchronization pipeline described in this study enables the generation of reusable datasets that can be extended to additional anatomical regions, pathologies, or scanning techniques. This modular structure allows future upgrades and the incorporation of broader datasets, potentially expanding the scope of immersive ultrasound training environments.

Future research directions may build upon this foundation in several ways. First, the development of structured learning tasks within the virtual environment could transform exploratory interaction into goal-oriented training modules. For example, the system could require users to reproduce specific probe positions, identify anatomical landmarks, or achieve predefined imaging conditions, with performance metrics automatically evaluated by the framework. Second, comparative studies assessing skill transfer between immersive-based training and conventional methods would provide further insight into the educational impact of immersive approaches. Expanding the dataset to include pathological cases and varied anatomical conditions would increase the range of available learning scenarios.

Several aspects of the current implementation merit acknowledgement as boundaries of the present study. The dataset was acquired from a single cadaveric specimen (a 65-year-old male) and therefore does not yet reflect the anatomical variability encountered in clinical practice, including differences in body composition, muscle architecture, and pathological tissue presentations. This constraint is an inherent characteristic of any initial dataset rather than a limitation of the framework itself: the acquisition and synchronisation pipeline described in this work is specifically designed to be repeatable, and its application to additional specimens, varied demographics, or pathological cases is straightforward within the same technical infrastructure. The 10 fps recording rate, while appropriate for the slow, controlled probe movements characteristic of MSK ultrasound and sufficient for the synchronisation architecture described here, may reduce temporal fidelity in more dynamic scanning scenarios (for instance, rapid survey sweeps or needle-guidance procedures), and higher frame rates should be considered when extending the pipeline to such applications. More broadly, quantitative characterisation of system performance, including synchronisation accuracy between video frames and motion capture timestamps, end-to-end rendering latency, and spatial registration error, was not performed in this study and constitutes a direction for future technical validation. The evaluation was conducted with a small convenience sample of ten participants, which precludes statistical generalisation of the usability findings; this is consistent with the exploratory scope of the study and with practices documented in comparable early-stage evaluations of medical XR systems. Finally, the SUS instrument measures perceived usability and subjective acceptance; no conclusions regarding skill acquisition or knowledge transfer are drawn from the data reported here, and outcome-based assessment remains a defined priority for subsequent work.

Taken together, the findings suggest that immersive XR environments are perceived as potentially useful contributors to musculoskeletal ultrasound education, particularly with respect to clarifying the relationship between probe manipulation and image interpretation. While further validation, including objective assessments of learning outcomes and larger-scale studies, is warranted, the present framework establishes a feasibility foundation for future developments in immersive, data-driven ultrasound training.