Impact of Audio Feedback on User Experience in Haptic-Visual Mixed Reality Pulse Palpation Training Environments

Abstract

Background: Mixed Reality (MR) environments rely on multimodal feedback to enrich sensory integration and realism, which enhances User Experience (UX). Prior studies have shown the benefits of haptic feedback in audio–visual MR medical training environments, but researchers have not fully examined how audio cues influence Haptic–Visual (HV) training environments. Methods: We built a high-fidelity MR medical training environment that synchronized visual, haptic, and audio of the human pulse. We conducted a between-subjects study with thirty novice participants who performed pulse palpation tasks in HV and Haptic–Audio–Visual (HAV) modalities. We employ a multidimensional UX evaluation by measuring task performance, presence, usability, and task workload to assess the impact of adding audio feedback in MR pulse palpation training environments. Results: Participants in the HAV modality performed tasks more accurately and reported stronger presence and higher usability. They did not report any significant increase in workload compared to the HV modality. Conclusions: Audio feedback improved perceptual coherence and enhanced UX in pulse palpation tasks. Our findings highlight the training value of integrating multimodal feedback in MR pulse palpation training systems and provide practical guidelines for designing more immersive and effective MR environments.

1. Introduction

In recent years, Extended Reality (XR) technologies have profoundly impacted professional training across various industries. XR encompasses Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), each offering unique ways to blend digital and physical environments to create immersive training experiences. VR fully immerses users in a computer-generated world, while AR overlays digital content onto the real world, and MR allows for interaction between physical and virtual elements in real-time [1]. The application of XR for training in professional settings comes from the concept of simulation-based training. By replicating real-world scenarios in a virtual environment, XR simulations offer hands-on experience without real-world consequences, making them invaluable in high-risk professions such as healthcare, aviation, and defense [2,3]. For instance, VR can simulate a surgical procedure, allowing practitioners to practice operations in a controlled setting without the risks associated with real patients [4,5]. Similarly, AR can overlay step-by-step maintenance instructions onto real-world machinery, guiding technicians through complex repairs with increased accuracy [6,7]. These immersive simulations help trainees develop technical skills, spatial awareness, communication, and teamwork capabilities.

A key aspect of XR’s effectiveness in training compared to other training methodologies is its ability to provide multimodal feedback—integrating visual, auditory, haptic, and even olfactory stimuli. This multimodal feedback significantly enhances the realism of XR environments, allowing trainees to experience a richer, more engaging environment that closely mimics real-world scenarios [8,9]. By responding to these diverse sensory inputs, trainees can develop more intuitive and natural responses, making the training process more effective and aligned with the complexities of actual professional environments [10,11]. Multimodal feedback enhances perceptual accuracy, immersion, and overall user satisfaction [12,13]. Research consistently demonstrates that multimodal interactions facilitate richer sensory integration, reduce cognitive workload, and improve task performance and accuracy [13,14].

Multimodal XR interfaces are especially valuable in distance learning and telemedicine, promoting natural interactions between users and virtual objects [13]. One prominent use case of multimodal interfaces in XR is medical training simulators, which create repetitive, immersive simulations of medical procedures using realistic visualization and haptic feedback. Medical procedures demand precision and accuracy to ensure minimal error. Training healthcare providers is time-intensive and requires significant instructor resources. Many aspects of healthcare training rely on experiential learning, where learners integrate haptic, audio, and visual feedback under direct supervision [15]. Research highlights the benefits of combining XR and haptics as effective training tools in medicine [16].

Pulse palpation, a fundamental diagnostic procedure, requires doctors to integrate information from haptic, audio, and visual feedback. Each feedback modality conveys unique information: auditory feedback provides blood oxygen levels, while haptic feedback informs pulse pressure and rate. Clinically, pulse assessment involves palpation of the radial artery, along with Electrocardiogram (ECG) and pulse oximetry for blood oxygenation monitoring. Traditional pulse palpation training lacks diversity in pulse simulations, limiting student exposure. Our review shows that existing pulse training simulators primarily focus on haptic feedback [17,18]. Some studies like Ulrich et al. and Kandee et al. integrated haptic pulse palpation training devices with a desktop VR interface [19,20]. This enables visual feedback of the hand during interaction and provides a Haptic–Visual (HV) training environment for pulse palpation. Our review also reveals that all existing pulse palpation training simulators omit integrating audio feedback of the pulse, thus reducing its training efficacy.

Integrating audio cues in HV MR pulse palpation training environment could potentially enhance temporal coordination, rhythmic synchronization, and perceptual realism, thereby enriching user interactions and experiences [21]. This study investigates how introducing audio feedback into HV MR pulse palpation training environments impacts the overall UX of the participant. Employing a multidimensional, user-centered framework, we systematically evaluate UX aspects, including performance, presence, cognitive load and usability. The Research Question (RQ) addressed in the paper is as follows:

What is the impact of integrating audio feedback into haptic-visual MR pulse palpation training environments on user experience?

This work offers two principal contributions. We design and develop a MR-based multimodal pulse palpation training environment incorporating HAV feedback. This environment enables the investigation of multimodal interactions under ecologically valid conditions. Second, we conduct a controlled comparative study employing a multidimensional UX evaluation to empirically assess the role of audio feedback within HV MR pulse training environments. By examining objective and subjective metrics across HV and HAV modalities, we provide systematic evidence of the perceptual and experiential benefits of audio integration. Results show that participants performed better and reported a higher presence and in the HAV modality.

2. Literature Review

2.1. Multimodal Interfaces in XR

Multimodal interfaces, combining visual, auditory, and haptic stimuli, are essential for improving user engagement, task performance, and overall satisfaction in diverse XR applications [8,22]. The integration of multiple sensory channels can effectively replicate real-world experiences, thus offering a higher sense of realism and immersion compared to unimodal interfaces. Visual feedback remains fundamental to XR systems, typically involving immersive graphics and augmented overlays that facilitate real-time interaction and situational awareness [23]. However, visual cues alone can be insufficient, particularly for tasks requiring fine motor skills. Thus, augmenting visual information with audio and haptic feedback enhances perceptual accuracy and reduces cognitive load [14].

Audio feedback is widely utilized in XR to enhance spatial awareness and user orientation within virtual environments. Spatialized audio can provide crucial directional cues and enhance realism, thereby improving user interaction and environmental understanding [24,25]. Research indicates auditory feedback significantly aids in tasks requiring precise localization, timing, or alert responsiveness [26]. Addition of haptic interfaces also significantly enrich XR experiences by providing tactile sensations that closely mimic real-world interactions. Various technologies such as gloves, force-feedback controllers, and wearable devices provide realistic touch and force sensations, improving user interaction and immersion [27]. Studies have demonstrated that haptic feedback can improve precision in tasks involving manipulation or interaction with virtual objects by replicating realistic sensations such as texture, resistance, and pressure [28,29].

Despite their advantages, multimodal interfaces face significant challenges. One primary limitation is sensory overload, where poorly synchronized feedback can negatively impact user performance and experience. Users can become overwhelmed, distracted, or experience cybersickness due to mismatched sensory inputs, highlighting the importance of careful feedback synchronization and modality selection [30,31]. Haptic fidelity also remains a substantial barrier, as current technologies fail to fully replicate the tactile experiences required for precise interactions, limiting the realism of XR environments [32].

Another limitation identified in the literature is the skewed research emphasis towards exploring the effects of adding haptics to Audio–Visual (AV) XR environments [32]. However, many tasks, such as pulse palpation, rely primarily on tactile interactions, and existing HAV studies do not fully replicate the dominance of haptic feedback in these contexts. Integrating audio cues into HV environments could enhance user experience due to the strong correlation between audio and haptic feedback [33].

2.2. Techniques to Simulate Human Pulse

This section reviews various techniques used to simulate human pulse in existing pulse simulators. We analyze the advantages and disadvantages of each technique and discuss the feasibility of integrating these techniques into an XR simulator.

Human Patient Simulators (HPS), developed in the late 1900s, facilitated teaching cardiovascular hemodynamics and advanced clinical skills [34]. Utilizing hydraulic-based HPS, authors in [35] created a scenario integrating arterial pulses and arrhythmia to explain their relationship with cardiovascular hemodynamics. Similarly, Liu et al. designed an intelligent pulse simulation with modules mimicking the mechanical features of the human circulatory system [36]. Yang et al. developed a pulse simulator generating age-specific pulse pressure waveforms by varying the augmentation index of the pulse [37], expanding the versatility of hydraulic-based HPS.

Researchers have also explored force-feedback devices and mechanical systems for simulating human pulse. Ullrich et al. introduced a software-based haptic pulse simulation using a force-feedback device (Phantom Omni), offering dynamic arterial pulse behavior [19]. Additionally, Kandee et al. used the Phantom Omni to simulate arterial pulses, and Coles et al. simulated femoral pulse using a 3-DOF Falcon device in a closed-loop configuration [20,38]. Mechanical systems such as cam-followers [39] and pneumatic actuators [18] have also been employed to build compact pulse simulators.

Electromagnetic and smart material actuators provide versatility and easy customization compared to other techniques. Koo et al. utilized Magneto-Rheological (MR) fluids to simulate different pulse waveform patterns by controlling magnetic fields through Pulse Width Modulation (PWM) [40]. Luo et al. demonstrated the feasibility of high-frequency Linear Resonance Actuator (LRA) vibrotactile stimuli for rendering fine temporal profiles of pulse waves [17].

Table 1. A summary of Pulse Simulation techniques, strengths, and limitations.

Technique	References	Strengths	Limitations
Hydraulic Equipment	[35,36,37]	Provides detailed pulse waveforms and integrates cardiovascular hemodynamics	Bulky systems, limited portability
Force-Feedback Devices	[19,20,38]	Dynamic and realistic pulse feedback	Requires specialized devices (e.g., Phantom Omni), limited to certain pulse types
Mechanical Systems	[18,39]	Compact and simple designs	Scalability issues, only simulates specific pulse types
Smart Materials	[40]	Customizable pulse waveforms, versatile	Bulky systems, complex setup
Electro-magnetic Actuators	[17]	Portable, affordable, provides fine control of pulse waveforms	Less established in clinical settings compared to other methods

summarizes the various techniques used in the literature to simulate the haptic feedback of human pulses. It also entails the advantages and disadvantages of each technique.

Several studies have explored the integration of visual and haptic feedback in XR simulators for medical training, specifically for pulse palpation. For example, Ulrich et al. developed a pulse simulation system that uses a force-feedback device, Phantom Omni, in conjunction with a desktop VR interface to deliver visual feedback of hand interactions during pulse assessment [19]. Similarly, Kandee et al. and Coles et al. used haptic devices to simulate pulse feedback, with visual displays that enhance the user’s perception of pulse location and intensity [20,38]. These approaches provide valuable haptic and visual feedback for users, making pulse simulation more interactive and realistic compared to traditional training tools.

However, despite these advancements, existing pulse training simulators still lack audio feedback, which is a vital component of comprehensive pulse assessment. In real medical settings, audio cues of pulse such as pulse oximetry signals contribute essential information about a patient’s condition. Without this audio dimension, current XR simulators fall short of fully replicating the sensory experience of pulse palpation, which may limit their effectiveness in training healthcare professionals. Addressing this gap, our work aims to develop a pulse simulator that integrates all three modalities (HAV) of human pulse to create a high-fidelity MR training experience.

2.3. Evaluating UX in Multimodal MR

Recent advances in MR technology make it extremely critical to understand and optimize UX. UX evaluation ensures users achieve optimal engagement, immersion, comfort, and effectiveness. Given MR’s complexity, UX evaluation must integrate multiple dimensions, encompassing objective metrics and subjective metrics. Objective metrics include physiological, behavioral, and performance-based metrics, and subjective metrics include measures of user perception, comfort, immersion, and emotional impact [41,42].

2.3.1. Objective Dimensions of UX Evaluation

Objective dimensions of user experience evaluation focus on quantifiable, observable data that reflect users’ interactions and responses within MR environments. By employing physiological signals, behavioral patterns, and task performance indicators, these metrics offer unbiased insights into how users engage with and react to mixed reality systems. Their empirical nature enables developers and researchers to identify usability bottlenecks, cognitive load, and interaction efficiency, providing a strong foundation for data-driven design improvements.

Physiological metrics measure users’ cognitive load, stress, and emotional responses through indicators such as Heart Rate Variability (HRV), Electrodermal Activity (EDA), and Electroencephalography (EEG). These provide real-time insights into users’ subtle, often subconscious reactions to MR experiences [43,44]. Despite their potential, physiological metrics pose practical challenges like interference with head-mounted displays, user discomfort from sensors, and complex data interpretation [45].

Behavioral metrics also objectively capture user interactions within MR environments, including gaze direction, movement amplitude, spatial navigation, interaction patterns, reaction times, and gesture analysis. These metrics offer valuable insights into how users physically engage with MR systems, aiding the identification of design strengths and weaknesses. They are advantageous due to their real-time data collection and minimal user intrusion, providing comprehensive and actionable data for UX improvements [41,46].

Lastly, performance metrics evaluate users’ ability to execute tasks effectively within MR environments. Typical measures include task completion time, error rates, task accuracy, efficiency, and overall task effectiveness. These metrics directly reflect MR systems’ practical utility and usability, guiding iterative improvements and refinements in MR designs [42,47].

While the objective dimension provides measurable insights into users’ physical and behavioral responses, it offers only part of the picture. To fully understand user experience in MR, it is essential to consider subjective dimensions, which capture users’ perceptions, feelings, and personal evaluations.

2.3.2. Subjective Dimensions of UX Evaluation

Subjective dimensions capture the personal, introspective side of user experience in MR, emphasizing users’ self-reported perceptions and emotions. These evaluations explore how users feel about the system’s usability, comfort, immersion, and emotional impact. They are crucial for understanding satisfaction and engagement levels that objective metrics alone cannot fully capture.

Subjective evaluations of usability, ease of use, comfort, and ergonomics significantly influence user satisfaction. Instruments like the System Usability Scale (SUS) [48], NASA Task Load Index (NASA-TLX) [49], and User Experience Questionnaire (UEQ) [50] quantify user perceptions effectively, providing rich insights into usability issues and facilitating targeted improvements in MR system design. Users’ comfort and ease of use can dramatically affect the adoption and continued use of MR technologies. Factors such as physical discomfort, cognitive load, and interaction complexity directly correlate with overall satisfaction and usability [51].

Immersion and presence represent users’ psychological and sensory engagement with virtual environments. These subjective dimensions are crucial in MR, influencing user satisfaction, performance, and overall experience quality [52]. Immersion reflects the extent to which users feel surrounded by virtual stimuli, while presence describes their psychological sense of “being there” within the virtual environment [46,53]. Standardized tools such as the Presence Questionnaire (PQ) and Immersive Experience Questionnaire (IEQ) effectively evaluate these subjective dimensions [54].

Evaluating emotional responses, enjoyment, and simulator sickness is vital to understanding MR’s psychological impact as it influences user interest, attention, satisfaction, and retention. Various scales, including the Simulator Sickness Questionnaire (SSQ) and Positive and Negative Affect Schedule (PANAS), help measure emotional and psychological responses, facilitating targeted design adjustments for improved emotional comfort and user safety in MR environments [55,56]. Additionally, research highlights the correlation between emotional responses and user performance, indicating that positive emotional states foster enhanced interaction in XR [57]. Hence, a multidimensional approach integrating both objective and subjective UX metrics provides comprehensive insights essential for enhancing MR experiences.

3. Research Methods

This research investigates the impact of integrating audio feedback into an HV MR pulse palpation training environment. We evaluate the effectiveness by focusing on its effects on UX. We employ a quantitative research methodology that combines a controlled study with empirical user evaluation. As part of this work, we design and develop an MR-based medical hospital environment for performing the pulse palpation task. This environment is designed to simulate the haptic, audio, and visual feedback of the human pulse to provide users with a realistic environment for performing the tasks.

To evaluate the role of audio feedback, we conduct a between-subjects user study comparing two modalities: HV and HAV. In the HV modality, participants receive haptic and visual feedback of the human pulse. In contrast, in the HAV modality, they receive haptic, audio, and visual feedback while performing tasks in the MR environment.

We employ a multidimensional approach to evaluate the UX of the participants by integrating both objective and subjective dimensions. The objective dimension consists of performance metrics, and the subjective dimension consists of presence, usability, and task workload experienced by the participant. This research helps answer the research questions by analyzing and performing statistical tests, such as a t-test, on the objective performance and perception data. The findings contribute to a deeper understanding of how audio feedback enhances haptic-dominant interactions in an MR environment.

4. Development of HAV MR Pulse Palpation Training Environment

This section details the development of the MR training environment, which integrates haptic, audio, and visual feedback of the pulse. The environment development follows multimodal interaction principles, ensuring real-world like sensory feedback experience. Figure 1 illustrates the system architecture, which consists of software and hardware components that communicate via a messaging protocol to synchronize multimodal feedback in real-time.

4.1. MR Environment Hardware—Haptic Feedback

This section details the development of the hardware component of the MR pulse palpation training environment that is designed to provide the haptic feedback of the patient. The haptic feedback system uses a 1-degree-of-freedom (DOF) force-feedback device. We implement the system using a Voice Coil Actuator (VCA) enclosed in a hand mannequin and the VCA is controlled by a Raspberry Pi 4, allowing for precise force modulation as shown in Figure 2. VCAs are widely recognized for accurate mechanical feedback and are classified as medium-fidelity haptic devices within Breitschaft et al.’s framework [58].

Table 2. Voice coil actuator parameters.

Technical Parameters	Value (Unit)
Intermittent Force @ 30 KHz
70% duty cycle, 2 W	10 N
Max Stroke	6 mm
Coil Clearance per side	0.32 mm
Coil Assembly Mass	5 gm
Body Mass	7 gm
Coil Resistance	1.9 Ω
Coil Inductance @ 1 KHz	65 μH
Max Continuous Power	4.0 W

outlines the technical specifications of the VCA used for this research.

We use the Gaussian approximation of pulse flow as input to the VCA to actuate the radial pulse [20]. The hardware system actuates pulses within the 0.2–0.9 N force range. The implementation closely simulates a patient’s wide range of pulses in trauma.

To validate the accuracy of the haptic feedback generated by the VCA, we compute the Force Feedback Error ($F{F}_e$), which represents the percentage variance between input/actuated haptic feedback and the actual generated haptic feedback Equation (1).

$$F{F}_e={\displaystyle \frac{|F{F}_{input}\left(N\right)-F{F}_{actual}\left(N\right)|}{F{F}_{input}\left(N\right)}}\ast 100$$

(1)

$F{F}_{input}$ refers to the input commanded force(N) and $F{F}_{output}$ refers to the actual force produced (N) by the VCA. We measure the produced force ($F{F}_{output}$) of the VCA using the FSR 04 sensor by Tekscan Flexiforce. We observed a 4.2% error in generated haptic force feedback, indicating a high level of accuracy.

4.2. MR Environment Software—Visual and Audio Feedback

We developed an MR-based training environment that simulates the visual and audio feedback of human radial pulses. We choose MR for developing our training environment as fully immersive environments restrict physical interactions, whereas MR environments allow embodiment and virtual space navigation [59]. The MR application uses Unity and Microsoft’s Mixed Reality Toolkit (MRTK). Visually, the user is immersed in an MR-based trauma bay with an interactive patient, an X-ray display, and a multi-parameter pulse monitor, as shown in Figure 3. All the assets are open source and available on the Unity asset store. We implemented the MR environment so that the patient’s arm builds on the hand mannequin of the hardware component using the spatial mapping feature of MRTK. As a result, when the participant touches the patient’s wrist to investigate the radial pulse, they will experience haptic pulse feedback. Participants engage with the system as first-person characters, undertaking various training tasks related to pulse palpation.

In the MR environment, the users can also hear the patient’s audio feedback on the pulse from the multi-parameter pulse monitor. The auditory feedback of the pulse relates to the ${\mathrm{SpO}}_2$ of the pulse, where a decreased audio pitch indicates decreased oxygen reaching peripheral tissues and vice-versa [60]. We obtain the audio frequency associated with the patient’s ${\mathrm{SpO}}_2$ using the function derived in [26] and stated in Equation (2).

$$Frequenc{y}_{\left(Hz\right)}=37.128e^{0.0289\left({\mathrm{SpO}}_2(\%)\right)}$$

(2)

To implement the audio feedback in MR environment audio generation, we modify the pitch of a standard pulse oximeter audio signal so that it corresponds to the pulse being simulated. This technique is based on the relationship between the frequency and pitch of an audio signal [61]. Frequency is a physical parameter of the sound, and pitch is what an individual perceives. Similar to the haptic feedback, we calculate the Percentage Error in generating audio feedback ($A{F}_e$) to evaluate the accuracy of the auditory feedback generated. It represents the percentage variance between the input audio frequency and the actual generated audio frequency Equation (3).

$$A{F}_e={\displaystyle \frac{|A{F}_{input}-A{F}_{actual}|}{A{F}_{input}}}\ast 100$$

(3)

Here, $A{F}_{input}$ refers to the input commanded audio frequency(Hz) and $A{F}_{output}$ refers to the actual audio frequency produced (Hz). We use Spectroid, an open-source audio analyzer, to measure the frequency of the audio signal generated by the Unity application. The calculated value for $A{F}_e$ is 0.2%, confirming high-fidelity audio simulation.

4.3. Integrating HAV Feedback in the MR Environment

We then use the MQTT protocol to facilitate the connection between the hardware component and the XR application. MQTT is a lightweight messaging protocol that consists of a topic and a message. The topic consists of a unique string value, where all messages using that string value communicate on the same channel [62]. The environment starts the pulse palpation simulation by taking input from the user in the form of voice commands or hand interactions. When it receives this, the MR environment sends a message through MQTT to Raspberry Pi 4, which is also the MQTT broker.

For each pulse actuated on the VCA, the Raspberry Pi 4 (hardware component) sends a message back to the MR pulse simulator to play the audio feedback and modify the visual feedback of the pulse. This helps synchronize the haptic, audio, and visual feedback of the pulse. Figure 4 depicts the entire workflow and tech stack used to develop the environment. Baseline latency for Microsoft HoloLens 2 applications averages 55 ms [63]. In the HV scenario, adding force feedback increased latency by 30 ms due to MQTT network latency and VCA response time, while the HAV scenario, which included both audio and force feedback, added approximately 33 ms. To achieve this latency, we employ multi-threading, where network communication, modifying and playing the audio signal, and updating the visual feedback run on parallel threads.

5. User Study Design

We perform a comparative user study to understand the impact of audio feedback while integrating multimodal (HAV) feedback in MR pulse palpation training environment on the participant’s performance and experience. The study is conducted using the developed HAV MR training environment. It is a comparative study between participants, in which the participants performed the tasks in one of the two environment modalities: HAV or HV. In the HV modality, the participants received haptic and visual feedback about the simulated pulse, whereas in the HAV modality, they received feedback through haptic, audio, and visual modalities. The objective of the study is to understand the impact of adding audio in an HV MR environment on the participant’s UX.

5.1. Participants

We conducted the user study with 30 novice participants (Female = 14 and Male = 16). The age of the participants ranged from 18 years to 37 years. Half of the participants did the study in the HAV modality, and the other half did the study using the HV modality. For this study, we considered only participants who were right-hand dominant to ensure consistency.

Table 3. Demographics of Participants in this study detailing the gender distribution, experience measuring human pulse, and experience with using XR and haptic devices (N = 30).

Characteristic	HV Scenario	HAV Scenario
	(N = 15)	(N = 15)
Gender
(Male:Female)	7:8	9:6
Measured Pulse
(Yes:No)	10:5	9:6
Used XR Device
(Yes:No)	13:2	14:1
Used Haptic Device
(Yes:No)	10:5	11:4

details the participant’s demographics in each study category. Most participants had some basic previous experience of measuring their own pulses and interacting with XR and haptic devices.

5.2. Study Scenario

This study focuses on the tasks of identifying changes in pulses and classifying pulses based on their perceived pulse force strength. We base this on the monitoring requirements of the patient’s pulse. Based on the pulse force, there are five classes of pulses—Strong/Bounding, Normal, Weak, Thready, and No Pulse.

Table 4. The pulse parameter ranges for pulses in each category.

Pulse Category	Heart Rate (bpm)	Blood Pressure (mmHg)	SpO2 (%)	Force Range (N)	Audio Range (Hz)
Strong	95–120	120–130	90–95	0.83–0.9	545–620
Normal	65–95	100–120	95–99	0.7–0.83	620–715
Weak	45–65	70–100	85–90	0.48–0.7	470–545
Thready	35–45	50–70	80–85	0.34–0.48	405–470
No Pulse	20–35	30–50	70–80	0.2–0.34	310–405

depicts the range of the various pulse parameters for each category and the force and audio range for simulating the pulse of each category.

For the study, we implement an HAV scenario, where the patient’s ${\mathrm{SpO}}_2$ progressively declines from 99% to 70%. Correspondingly, the pulse rate decreases from 120 bpm to 25 bpm and pulse pressure from 130 mm Hg to 30 mm Hg [64]. To realize a smooth transition, we implement 72 different pulses, with each having unique values of ${\mathrm{SpO}}_2$, pulse rate, and pulse pressure, as illustrated in Figure 5. The design of the transitions between two pulses took into account the Just Noticeable Difference (JND) thresholds for haptic and audio feedback [65] to make the simulation realistic. This range of parameters covers all five categories of the pulse.

5.3. Procedure and Tasks

The participants start with filling out a pre-study demographic questionnaire at the start of the study. Next, all the participants were presented and informed about human pulses, their attributes and the tasks they are required to do. They then wear the Microsoft HoloLens 2 headset and interact with the MR environment to familiarize themselves with the environment. As part of their training, they interact in the, learn to palpate radial pulses and also experience one pulse of each category. The participants then perform two tasks as part of the study, with each task focused on gathering specific performance metrics.

In the first task, the participants experience a scenario of a patient in the MR environment as shown in Figure 6 with continuous pulse changes, transitioning from one category to another. As part of this task, the participants were required to identify and report all the changes observed in the pulse. We collect the true positive (changes identified correctly) and false negatives (missed changes) detections by each participant. Using these values, we calculate sensitivity of the participant towards identifying changes among the pulses using Equation (4).

$$Sensitivity={\displaystyle \frac{TruePositive}{TruePositive+FalseNegative}}$$

(4)

In the second task, we randomly simulate five test pulses, each belonging to a different category, and the participants are required to classify them and then measure their pulse rates. They had a virtual clock available in the MR environment to help them measure the pulse rate.

Using the data collected here, we calculate the participant’s accuracy in classifying the pulse and the error rate in measuring the pulse rate.

In the end, the participants filled out the surveys about the presence they felt while performing the task in the environment, the workload experienced while performing the tasks and the usability of the system. We then interviewed the participants briefly about their experience interacting with the multimodal MR environment and what helped them perform the task better. All the data collected through the study helps us to understand the impact of audio feedback in HAV MR pulse training environments on the overall UX of a participant.

5.4. Materials and Tools

The study is conducted on Microsoft HoloLens 2 headset. The physical study setting is an empty room with the hand manikin on the table. As discussed earlier, the virtual objects are spatially mapped to render the virtual patient above the mannequin. A Study personnel is in the room with the participants to guide them in transitioning from one task to another and collect the evaluation data.

The study is a between-subjects comparative study. The participants were randomly assigned a scenario for the study. As stated in Section 3, we employ the multidimensional definition of UX for our research. We consider objective task performance metrics like sensitivity, accuracy, and error rate, and subjective perception metrics like presence, usability, and task workload to wholistically evaluate UX of a participant. We do not collect physiological data as they are effective in anxious scenarios like responding to a threat and [66,67] argue against using it static simulations. Using these metrics, we infer the following about the overall UX of the participants.

5.5. Hypothesis

The study hypotheses for the study are designed to address the research question defined in Section 1. They are as follows:

H₀: 

Participants’ objective dimension of UX, calculated through their task performance scores, will be the same in both HAV and HV modalities.

${\mathbf{H}}_{\mathbf{0}}^{\prime }$: 

Participants’ subjective dimension of UX, calculated through their reported presence, workload and usability scores, will be the same in both HAV and HV modalities.

5.6. Data Collection and Statistical Analysis

We collect the task performance metrics data manually by the study personnel while the participants perform the tasks during the study. To collect the subjective data, we use the following standard questionnaires: SUS survey [48], NASA TLX [49], and iGroup Presence survey [68].

To validate the hypotheses, we performed the t-test on the participant’s performance and perception scores for both modalities. We later discuss the qualitative data collected through end interviews to understand the underlying factors impacting the participant’s experience while interacting with the MR environment.

5.7. Ethical Considerations

All the participants consented to take part in the study. The Institutional Review Board approved the study design and data analysis.

6. Results

In this section, we discuss the observed results from the user study conducted with 30 participants (15 HV and 15 HAV).

6.1. User Performance: Objective Dimension of UX

6.1.1. Task 1: Sensitivity of Participants Towards Pulse Changes

In the first task, the participants experience a realistic scenario of pulse changes. As described in Section 5, we simulate 72 pulse changes. Figure 7a visualizes the distribution of the the total number of correct changes observed by the participants during the study in each scenario. From the plot, we observe that participants detected more pulse changes in the HAV modality of the MR training environment. This suggests that adding audio cues enhanced users’ sensitivity to pulse variations.

6.1.2. Task 2: Accuracy of Classifying Pulses and Error Rate for Calculating Pulse Rate

In the second task, participants measured the pulse rate which is an essential skill while palpating pulse. Figure 7b visualizes the error rate (in percentage) of the participants while calculating the pulse rates in both the study scenarios. From the box plot, we observe that most of the participants performing the study in the HAV modality have lower error rates than the participants in the HV study scenario. Adding audio cues helped maintain a steady rhythm, which is crucial for accurate pulse rate assessment.

In this task, the participants were also required to classify the five test pulses according to the perceived strength. Figure 7c shows the distribution of the participant’s accuracy scores. The box plot shows that participants in the HAV modality achieved higher accuracy in classifying pulses into the five categories. This improvement indicates that audio cues helped participants better distinguish pulse types, which is crucial for rapid, accurate assessments in medical settings.

6.1.3. Statistical Analysis

We perform a t-test to statistically validate the results observed. For sensitivity and error rate, we observe a p-value < 0.001 and for accuracy, we observe a p-value < 0.01. This states that the difference between the performance scores of participants in the HAV modality and HV modality is statistically significant. Hence, the performance of the participants in the MR environment with HAV feedback modalities was significantly better than the participants performing the task in the MR environment with only HV feedback modality.

6.2. User Perception: Subjective Dimension of UX

6.2.1. Presence

To calculate a user’s presence, we use the standard presence survey developed by iGroup, which consists of fourteen questions. These questions are distributed over four parameters: General Items, Spatial Presence, Involvement, and Realness. Participants felt a stronger sense of presence in the HAV modality compared to the HV modality, as shown in Figure 8a. Hence, adding audio cues enhances users’ immersion, aligning the simulation more closely with real-world scenarios where HAV feedback coexists.

6.2.2. Task Workload

Task workload, measured via NASA TLX [49]. It classifies workload into six subjective categories: Mental Demand, Physical Demand, Temporal Demand, Performance Effort, and Frustration. For each category, we collect data on a 7-point Likert scale (1-very low, 7-very high). The overall workload experienced by the participants in both scenarios can be classified as Somewhat High. Figure 8b show that a participant experiences more workload in the HAV modality than in the HV modality. We discuss this increase in mental workload in the next section. We observe that the participant experienced more mental workload in the HAV modality, and we will explore this in detail in Section 7.

6.2.3. Usability

Usability, assessed through SUS Survey [48], consists of 10 questions and is answered on a 5-point Likert Scale. The questions are a combination of positive and negative questions ordered alternatively. The calculated average SUS score in HAV modality is 82, and in HV modality, it is 76.67 as depicted in Figure 8c. The higher usability in the HAV modality implies that audio feedback positively contributes to intuitive interactions, likely by reinforcing tasks with sensory feedback closer to real medical settings.

6.2.4. Statistical Analysis

We perform t-test on the calculated presence, workload and usability scores to statistically validate the results observed. For overall presence scores we observe a p-value < 0.001, making the increase in presence felt by participants in the HAV modality statistically significant. From t-test on usability scores we get the p-value < 0.05, and hence the usability of the MR environment with HAV feedback was significantly higher than the MR environment with just HV feedback. Lastly for the overall task workload score, we calculate the p-value = 0.563. Though there is an increase in the workload experienced by the participants in the HAV modality of the MR environment, it is not statistically significant increase.

6.3. Overall UX

The combined analysis of objective and subjective metrics presents a comprehensive understanding of the overall UX across both study conditions. Participants interacting with the MR training environment with HAV feedback modalities consistently outperformed those interacting the HV feedback modality across all performance tasks, including sensitivity, pulse rate calculation, and classification accuracy. These improvements were statistically significant, indicating that the integration of audio cues enhanced the users’ ability to perceive, interpret, and act on pulse signals in a simulated setting.

From a subjective perspective, the HAV modality also demonstrated stronger user perceptions across key UX dimensions. Participants reported significantly higher presence scores, indicating a greater sense of immersion and realism. Usability scores, as measured by the SUS, were also significantly higher in the HAV modality, suggesting that users found the multimodal system more intuitive and supportive during task execution. Although the overall task workload in the HAV modality was slightly higher, particularly in the domain of mental demand, this increase was not statistically significant. This suggests that while the addition of audio cues introduced minor cognitive effort, it did not substantially hinder the overall user experience. This highlights the potential of well-integrated multimodal design in XR training applications. Together, these findings indicate that the MR pulse training environments with HAV feedback modality resulted in a better UX without compromising user manageability.

7. Discussion

7.1. Impact of Audio Feedback in MR-Based Pulse Training Environments

The results of this study indicate that augmenting a HV MR pulse training environment with audio feedback can enhance UX across both objective and subjective dimensions. Participants in the HAV modality showed significantly higher performance in detecting pulse changes, calculating pulse rates, and classifying pulse strength. These improvements suggest that the additional auditory channel supported users’ perceptual processing during complex, time-sensitive tasks.

Studies in the literature have observed that adding additional modalities can impose extra cognitive load, especially if the sensory inputs are not well integrated or become overwhelming [30,31]. In some cases, multisensory feedback has even been found to distract users from the core objectives of the task, particularly when the feedback is incongruent with the task context or unintentionally competes for attention [69].

Although participants in the HAV modality reported slightly higher mental demand on the NASA TLX, the increase was not statistically significant. This suggests that the additional sensory channel did not impose excessive cognitive burden, particularly when the feedback was perceived as task-relevant. In our study, the audio cues were semantically and temporally aligned with pulse changes, and participants selectively relied on them based on the nature of the signal. During the post-study interview participant’s stated that they relied more on audio cues for detecting weaker pulses and haptic feedback for stronger ones—indicating adaptive sensory prioritization. This aligns with the Working Memory Theory (WMT) [70], which posits that users dynamically allocate cognitive resources based on stimulus salience and task demands. The findings demonstrate that, when carefully designed, audio feedback can serve as a beneficial cognitive aid to the user.

7.2. Insights Towards Designing HAV MR Training Environments

These findings, while grounded in the context of medical training, have broader relevance for other domains that rely on complex, high-stakes decision-making in immersive environments. In aviation and aerospace training, for example, pilots rely on auditory and haptic cues to maintain situational awareness. Incorporating synchronized audio feedback into MR flight simulators could improve anomaly detection and procedural adherence. Similarly, in industrial safety, MR systems enriched with audio-haptic feedback may support quick fault identification and safe operation in high-risk settings. Military simulations could use such feedback to reinforce spatial coordination and tactical awareness, while educational applications could use it to scaffold complex concepts and improve retention through multisensory engagement.

What emerges across these applications is a core design insight: multimodal feedback is most effective when it is contextually relevant and cognitively manageable. This study also reinforces the value of combining objective and subjective UX measures to evaluate such designs, showing that enhancements in performance and perceived quality must be balanced with the mental demands they introduce. Designers must be careful not to assume that increasing sensory input automatically improves experience; instead, modalities should be layered in ways that align with task structure and user cognition. When thoughtfully implemented, audio feedback can be a powerful tool in MR system design—not only enhancing realism and usability, but also extending the applicability of immersive environments across a range of real-world domains.

8. Conclusions

This research demonstrates the value of integrating audio feedback into HV MR environments to enhance UX. We created an ecologically valid platform, a MR pulse palpation training environment, to investigate multimodal sensory integration. Our empirical findings show that participants in the HAV modality outperformed those in the HV modality across all performance metrics, including sensitivity to pulse changes, classification accuracy, and pulse rate estimation. Subjective evaluations further revealed significantly higher presence and usability scores among participants performing the tasks in the HAV modality, with only a modest increase in workload.

These results provide strong evidence that audio feedback enhances perceptual clarity, improves task performance, and deepens immersion in haptic-dominant MR training environments. Importantly, we address a gap in the literature by focusing on integrating audio cues in HV contexts, where tactile and temporal signals are critical.

In summary, this work establishes a technical foundation for the development of HAV MR training environments, showing that multimodal feedback improves both objective performance metrics and subjective experience. Our work underscores the importance of modality synergy in XR system design and offers practical guidance for developers aiming to build more effective, intuitive, and immersive multimodal MR environments. In our future works, we plan to extend this work and investigate the impact on expert healthcare professionals.